Managing log files is becoming increasingly harder with growing amounts of data and differing file formats. Giovanni Bechis, in his upcoming talk at LinuxCon Europe, describes a solution using the ELK stack (ElasticSearch, Logstash, Kibana), which he says let’s you easily collect, parse, and manage log files from different sources.
We talked with Bechis, a Software Engineer at SNB S.r.l., a to learn more about how ELK can be used to aggregate any kind of data in a productive way.
Giovanni Bechis: ELK stands for Elastic Stack, and it is composed of Elasticsearch, a search engine; Logstash, a log collector; and Kibana, a powerful web interface. The Elastic Stack makes searching and analyzing data easier than ever before by delivering actionable insights in real time from almost any type of structured and unstructured data source.
Linux.com: How does ELK make log management easier?
Giovanni: Log management is easier than before because unstructured log files are now saved on a non-relational database and can be analyzed through a web interface. Through Kibana, you can then create graphs and summarize important informations without writing complex programs.
It is easier to manage and merge, for example, log files coming from different email software (e.g., Postfix and OpenSMTPD) and you can look at log files in real time even if they are coming from different hosts. There are also no problems due to the management of very big text files.
Linux.com: Who should be using ELK and why?
Giovanni: Every system administrator that manages more than one server should switch to ELK; managing log files in a multi-server environment could be very complicated and analyzing data to provide fancy graphs could create lot of headaches without a good framework to work with.
In an ELK world, all log files have a similar structure (JSON); this way it’s easier to create programs to query log files.
Linux.com: What are some examples of data other than log files that ELK could be used for?
Giovanni: For some customers, we use ELK to analyze data coming from their management software to be able to detect possible problems on their warehouses. Salesforce and Microsoft use ELK to analyze events generated from their CRM.
Elasticsearch is used without the other parts of the stack as a very powerful search engine by many web sites, from Facebook to MSN, to the New York Times.
Linux.com: Are there other features you’d like to implement? If so, what?
Giovanni: I am working on software to easily create alarms or execute actions if a log entry matches an expression. It will be an open source alternative to “watcher” — an Elasticsearch BV commercial product. We are using it at work to detect anomalies on our email servers and XSS attacks on our hosting platform.
You won’t want to miss the stellar lineup of keynotes, 185+ sessions and plenty of extracurricular events for networking at LinuxCon + ContainerCon Europe in Berlin. Secure your spot before it’s too late! Register now.
