Up And Running With Logstash

logstash

I want to talk about Logstash, a new-ish tool (to me) for managing computer logs. Logstash can easily collect logs from multiple computers or instances, transfer them to a central computer for aggregation and can even be used to parse and search these logs for analysis as they are handled. A mouthful indeed. Logstash is open-source software and is written in JRuby so it runs in the JVM. Running on the JVM has various advantages such as easy of deployment and wealth of tuning expertise available.

For demonstration purposes, lets setup a simple system with two computers. All computers are running Ubuntu 14.04. The first computer, call it Hebat, will be generating messages. The second computer, lets call this one Ilos, will be receiving these messages.

Hebat, 192.168.1.42 –> Ilos, 192.168.1.43

Hebat
We will be using syslog on Hebat for message generation. In order to send these messages to Ilos, we need to define Ilos as the recipient in Hebat’s rsyslog.conf as follows:

# Sending all entries to this computer:port
*.* @@192.168.1.43:3514

Lets cycle rsyslog for this revision to take effect.

sudo /etc/init.d/rsyslog restart

First computer is ready!

Ilos
After installing Java (1.7 at time of post), lets grab the latest copy of Logstash (1.4.2 at time of post).

Logstash is a framework that uses a configuration file to define all operating parameters. These parameters are defined by intent and these intents are called plugins.

For the task at hand, we need to configure both an input and an output plugin. If we were interested in parsing these messages, we would do this by defining a filter plugin.

  input {
    syslog {
      type => syslog
      port => 3514
    }   
  }
  output {
    file {
      path => "/var/log/all_msgs.txt"
    }
  } 

For input, we are telling the Logstash runtime to expect/consume syslog-type inputs on port 3514. This port matches the rsyslog configuration just done in Hebat. Logstash can do this over both UDP and TCP.

For output, we are having all messages saved in a local file in the path specified.

Lets go ahead and start Logstash up with our configuration file.

bin/logstash agent -f ../logstash.conf

The second computer is ready!

Testing
This task is similar in nature to centralizing computer logs using rsyslog for multiple computers in a system. We may want to use Logstash instead because we can take many different actions on our logs. Likewise, Logstash can receive many different inputs. For more information on these, just see the Logstash Docs.

To test our configuration, we only need to tail the text file in Ilos:

mariotalavera@hebat:~$
mariotalavera@hebat:~$ logger "this is a message on hebat; hello"

If everything went well, our message will get written to file on Ilos instantly.

mariotalavera@ilos:~$ tail -f /var/log/all_msgs.txt

{"message":"this is a message on hebat; hello","@version":"1",
"@timestamp":"2015-027T09:48:11.000Z","type":"syslog","host":
"192.168.1.42","priority":13,"timestamp":"Jan 27 04:48:11",
"logsource":"hebat","program":"mariotalavera","severity":5,
"facility":1,"facility_label":"user-level","severity_label":
"Notice"}

Neat, but a little anticlimactic. To perform a better test, lets attempt to send more than a message thru and see what happens.

For this, I am going to be using a syslog generator, sysloggen. Another Java program. This one basically takes in a file with sample log messages (to write to syslog) and randomly writes, or attempts to write these, to syslog. It has expected options as destination to write to, how many messages to send at a time and the total number of messages to send. All this information is available by running with ‘-h’ flag. Using sysloggen is fairly simple.

Besides Java, the only thing we need to do is to create a file with samples of the syslog messages we want sysloggen to randomly select. Lets call this file sample.conf

"a11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"
"b22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
"c33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333"

For this test, I’ve decided to just put three 100 byte messages for sysloggen to pick from. This sounds like a reasonable amount of information to transmit. Also, by making each line different, we could see how random sysloggen is (or not).

To run test, first we truncate syslog and do a line count. Lastly, we run sysloggen with the following parameters:

-d 192.168.1.43 (destination)
-f samples.conf (file with sample messages to send)
-l (loops thru messages in sample file)
-n 100 (messages to send)

mariotalavera@hebat:~$ > /var/log/syslog
mariotalavera@hebat:~$ wc -l /var/log/syslog
0 /var/log/syslog
mariotalavera@hebat:~$ ./sysloggen -d 192.168.1.42:514 -f sample.syslog -l -n 10
0
102 messages sent
425000 messages per second 
mariotalavera@hebat:~$wc -l /var/log/syslog
101 /var/log/syslog 

In Ilos, we check line count before and after running sysloggen in Hebat,

mariotalavera@ilos:~$ > /var/log/all_msgs.txt
mariotalavera@ilos:~$ wc -l /var/log/all_msgs.txt
0 /var/log/all_msgs.txt
mariotalavera@ilos:~$ wc -l /var/log/all_msgs.txt
101 /var/log/all_msgs.txt

Much better. We just sent 101 messages from Hebat to Ilos using Logstash; instantly.

Logstash has many other features worth exploring. As seen here, getting up and running is fairly straight forward. Having Logstash up and running lays the groundwork for upcoming posts and will hopefully ease its adoption!

Advertisements

One thought on “Up And Running With Logstash

  1. Pingback: Is Logstash Eating My Logs? | Mario Talavera Writes

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s