ELK Stack Notes

Foreword

Preliminaries

You need to install Java 8 or later and set up JAVA_HOME. If you install something newer than Java 9, some Elastic 6.4.2 agents will complain of an unrecognized VM option (UseParNewGC). This option disappeared in Java 9 and you may safely ignore this error. You may also safely ignore error, "chmod: cannot access '/etc/default/logstash': No such file or directory' if you see it.

These notes assume you're running Ubuntu Server 18.04.1 Bionic (headless). If something else, your mileage may vary. For example, earlier versions of Ubuntu, that is, prior to Ubuntu 15 Vivid, did not offer systemd, so the systemctl commands below will have to be done otherwise (at very least you can pop open a shell and run them from the command line).

Here is where the various Elastic component binaries are installed by their Debian packages. This is interesting to know as you follow steps in the next section:

1. /usr/share/elasticsearch/bin/elasticsearch
2. /usr/sbin/nginx
3. /usr/share/logstash/bin/logstash
4. /opt/kibana/bin/kibana
5. /usr/share/filebeat/bin/filebeat

Download, shallow configuration and installation (launch)

^* Yes, Elasticsearch is up and running on tuonela:

russ@tuonela ~ $ systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-08-25 09:27:56 MDT; 1h 35min ago
     Docs: http://www.elastic.co
 Main PID: 8403 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─8403 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSI
           └─8467 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 25 09:27:56 tuonela systemd[1]: Stopped Elasticsearch.
Aug 25 09:27:56 tuonela systemd[1]: Started Elasticsearch.

^** Yes, I changed Elasticsearch's password to changeme for now.

Tutorial stopped! tol-eressea, my Plex Media Server, is running this version:

Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-48-generic x86_64)

This version does not support systemd services. I must run Filebeat from the command line.

Reach ELK stack from browser

The next thing to do is to see if Elasticsearch is up, using curl:

$ curl tuonela:9200
{
  "name" : "u-tiTM2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Ic1V1rBCR32KHh06-xjzZA",
  "version" : {
    "number" : "6.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "595516e",
    "build_date" : "2018-08-17T23:18:47.308994Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

We could also use a browser and we must use a browser to reach Kibana. A caveat first, however. When I attempt to reach Elasticsearch or Kibana via a browser from a host at work running either of my VPN clients, it sometimes doesn't work unless and until I open a Chrome incognito window, but sometimes it does work. (I'm clueless, but it probably has nothing to do with the VPN.)

In a browser, for Elasticsearch, I see the same thing as from curl. Note that if I make my browser address line http://tuonela, I just get Nginx' welcome message, which is exactly what I'd expect.

For Kibana, here's my browser address line: https://tuonela:5601

...and, here's what I see:

Draining data from ELK

In early experimentation and testing, it might be useful to know how to drain ELK without having to obliterate the installation and start over from scratch. This amounts to draining data out of Elasticsearch. There are a couple of ways, the right one is to use the curl command below, but I'm giving other details too.

1. Use an Elasticsearch API:

   # systemctl stop filebeat.service
   # curl -XDELETE localhost:9200/_all
   

2. Remove all the indices in Elasticsearch's data: # systemctl stop filebeat.service # rm -rf /var/lib/elasticsearch/nodes/indices/*

Appendix: Securing ELK

A common security precaution is to allow access to Elasticsearch from only localhost and to restrict access to Kibana by using Nginx as a reverse proxy to Kibana running on localhost. Use either HTTP authentication to restrict access to Kibana or use LDAP authentication from Nginx.

Here are some useful links:

Appendix: Random notes