Skip to main content

An Experiment with Filebeat and ELK Stack

ELK Stack is one of the best distributed systems to centralize lots of servers' logs. Filebeat is a log shipper that keeps track of the given logs and pushes them to the Logstash. Then logstash outputs these logs to elasticsearch. I am not going to explain how to install ELK Stack but experiment about sending multiple log types (document_type) using filebeat log shipper to logstash server.

So there is one server (agent) filebeat and apache http server installed on it. Agent is going to send syslogs, http access logs and http error logs to Logstash.

Here is the contents of the /etc/filebeat/filebeat.yml config file:

There is one server (backend) logstash installed on it.
Here is the contents of the files located in the /etc/logstash directory:
conf.d/02-beats-input.conf
conf.d/10-syslog-filter.conf
conf.d/30-elasticsearch-output.conf

Comments

Popular posts from this blog

Find and replace with sed command in Linux

Find and replace feature is always handy. It can turn into a torture when it comes to change or delete a simple constant string in a text file. There is a handy tool in linux for doing these kind of tihngs. Actually sed is not a text editor but it is used outside of the text file to make changes.

Sending Jboss Server Logs to Logstash Using Filebeat with Multiline Support

In addition to sending system logs to logstash, it is possible to add a prospector section to the filebeat.yml for jboss server logs. Sometimes jboss server.log has single events made up from several lines of messages. In such cases Filebeat should be configured for a multiline prospector.
Filebeat takes lines do not start with a date pattern (look at pattern in the multiline section "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}" and negate section is set to true) and combines them with the previous line that starts with a date pattern.

server.log file excerpt where DatePattern: yyyy-MM-dd-HH and ConversionPattern: %d %-5p [%c] %m%n
Logstash filter: