Read the first item in this Table of Contents if you haven't been here before.
Table of Contents
- The ELKBeats Stack: Sounds Like a Good Idea ...
- The ELKBeats Stack: the Ground Work
- The ELKBeats Stack: L is for Logstash
- The ELKBeats Stack: E is for Elasticsearch
- The ELKBeats Stack: K is for Kibana
- The ELKBeats Stack: Getting E, L, and K to play nice together
- The ELK Stack with Beats: Feeding Logstash with Beats (Insecure - so far)
- The ELK Stack with Beats: Securing the Beats-to-Logstash Connection
This is a rough introduction to the ELK Stack and associated software: it's a big subject and is divided into multiple blog posts. Those are listed in the Table of Contents above: jump about as you feel is correct, but you should probably read this introduction first. If you're trying to build the entire stack, you should read the whole damn thing in sequence, and note that none of the rest of them are going to work without the Ground Work. The process given here was valid in March of 2016 for Debian systems using the Debian packages, not the tarballs: elastic.co's process and packages seems to be extremely volatile: if you're using some other distro or are significantly removed in time I suspect the process I've outlined won't work for you.
Trying to get ELK running has been an education in packaging and documentation. The most significant problem I can articulate so far is that elastic.co (who provide all of these pieces of software) provide both tarballs and packages for a variety of platforms (which is great), but the installation instructions occasionally fail horribly right after "unpack the tarball" or "install package X" because they explain how to configure usually the tarball when the package behaviour is totally different. The reasons for this are sound: the package installation system supports the creation of an unprivileged user to run the associated binaries, and a full start-stop system (ie. control scripts for Windows or Linux's systemd), but this also means that the configuration and behaviour are significantly different between the tarball and package, to the point that if you're working with the one they're not describing, nothing works at all.
elastic.co has extensive and imperfect documentation at http://www.elastic.co/guide/ . There's extensive documentation elsewhere as well, but all of it caused me a great deal of pain, so - like any good open source author who believes in fragmentation - I'm going to write my own. This is an outline: installation and configuration will follow in further blog posts. Keep in mind I'm using Debian, and that what I'm describing is the current Debian experience.
I'm going to let elastic.co make the case for their product:
Logstash is an open source data collection engine with real-time pipelining capabilities. Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice. Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
While Logstash originally drove innovation in log collection, its capabilities extend well beyond that use case. Any type of event can be enriched and transformed with a broad array of input, filter, and output plugins, with many native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data.
Ouch - the documentation author drank the marketing koolaid. In plain English? Logstash is the information gatherer. You point it at logs and other data sources via its complex and unpleasant configuration, and it converts those sources to JSON and feeds it to ...
elastic.co says "Elasticsearch is an open-source search engine built on top of Apache Lucene, a full-text search-engine library." A bit more detail: "Elasticsearch is a real-time, distributed storage, search, and analytics engine. It can be used for many purposes, but one context where it excels is indexing streams of semi-structured data, such as logs or decoded network packets."
"Distributed," huh? Let's see if we can even get one instance running ... My experience was that it's just as tricky to configure as Logstash.
Kibana is (mostly) the good news of this stack. It's relatively small (only compared to the other packages) and easy to install. And it either works or it doesn't (one of my attempts to install it failed completely and I have no explanation of that). I doubt it's a piece of software you'd use with anything other than Elasticsearch.
With all three services running (and not yet afflicted with any data or actually doing anything!), an otherwise entirely unloaded virtual machine has less than half of its 1G of memory remaining available. Make sure any machine running an ELK stack is well equipped.
Once again quoting elastic.co's documentation:
The Beats are open source data shippers that you install as agents on your servers to send different types of operational data to Elasticsearch. Beats can send data directly to Elasticsearch or send it to Elasticsearch via Logstash, which you can use to enrich or archive the data.
For my purposes, and likely yours too, Filebeat is your likeliest starting point:
Filebeat is a log data shipper initially based on the Logstash-Forwarder source code. Installed as an agent on your servers, Filebeat monitors the log directories or specific log files, tails the files, and forwards them either to Logstash for parsing or directly to Elasticsearch for indexing.
Prepare yourself for terms like "harvester" and "prospector."
Continue to The ELKBeats Stack: the Ground Work, the next article in this series.
(This is the same Bibliography for all of the "ELKBeats Stack" articles.)
- https://www.linode.com/docs/databases/elasticsearch/webserver-logs-with-elk-stack ... this is an excellent set of instructions that's significantly out-of-date (old URLs/addresses), which was nevertheless my main source of information
- http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html (with the caveat that as of 2016-03, my instructions are more accurate than theirs ...)
- Getting Kibana Up and Running
- Elasticsearch Getting Started
- Elasticsearch Reference >> Installation
- Elasticsearch Repositories (at elastic.co)
- Getting Started with Logstash
- Logstash Repositories (at elastic.co)
- How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04, Digital Ocean's uneven guide to this same subject, occasionally helpful but big on "install this" and short on "understand"