Monitoring VMWare ESXi with the ELK Stack

Monitor anything; this is what fills your head after playing with Elasticsearch, Logstash and Kibana.

ELK Stack, meet VMWare Server.  Since VMWare’s ESXi runs on some Linux kernel, it shares the logging facilities we’re familiar with on Linux systems.  Going about forwarding syslog messages to a remote box is a bit different thou.  Luckily, VMWare’s Knowledge Base is very thorough if you know where to look.

1. Enable SSH
This process needs to be done from the command line so ssh needs to be enabled on the server.  This can be done from vSphere Client.
2. Check syslog configuration
~ # esxcli system syslog config get
   Local Log Output: /vmfs/volumes/524f647f-31ac7e75-b467-08606ed69d08/systemlogs
   Local Logging Default Rotation Size: 1024
   Local Logging Default Rotations: 8
   Log To Unique Subdirectory: false
   Remote Host: 
If Remote Host: means there is no syslog forwarding set.
3. Enable Syslog forwarding
~ # esxcli system syslog config set --loghost='tcp://192.168.1.37:3514'
4. Re-checking syslog configuration
~ # esxcli system syslog config get
   Local Log Output: /vmfs/volumes/524f647f-31ac7e75-b467-08606ed69d08/systemlogs
   Local Logging Default Rotation Size: 1024
   Local Logging Default Rotations: 8
   Log To Unique Subdirectory: false
   Remote Host: tcp://192.168.1.37:3514
5. Restarting syslog
~ # esxcli system syslog reload
At this point, syslog has been restarted, however, tcp is blocked by firewall by default.
6. Checking host firewall
~ # esxcli network firewall get
   Default Action: DROP
   Enabled: true
   Loaded: true
7. Disabling firewall
~ # esxcli network firewall set -e=false
8. Reloading firewall
~ # esxcli network firewall refresh
It would be wise to just let TCP traffic thru firewall but I had difficulties doing so.  If you are on an internal network, you may be ok.
9. Re-checking firewall
~ # esxcli network firewall get
   Default Action: DROP
   Enabled: false
   Loaded: true
This is all the changes that need to be done on ESXi.  You can log out now.
On Logstash conf file, just add an input of type tcp matching the port defined in host.
10. Editing logstash conf file
  tcp {
    port => 3514
  }

Restart Logstash and let it warm up. If everything went well, we should see events trickling into Elasticsearch. Better yet, lets point a browser to Kibana and see them instead!

Screenshot 2015-05-15 at 6.16.31 PM

Not too bad.  From here, the next logical step would be to create a custom mapping for EXi host and build a dashboard on top of this.  Neat.  This pretty much settles the next post.

Advertisements

8 thoughts on “Monitoring VMWare ESXi with the ELK Stack

  1. Can you elaborate on the logstash config. I’m new to this and I’m not seeing anything in logstash.log when I send a test message from esxi

  2. Pikmin, Niklas,

    I just double-checked my settings and they are exactly as shown on post.

    MORE IMPORTANTLY, I just realized link to KB article was not showing. So sorry. I have fixed in post and provide here for your review: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003322

    You could also test by sending TCP traffic to Logstash from anywhere else. Logstash usually behaves very well. The ESXi configuration is more intricate, however.

    Good luck,

  3. Thanks for getting back to us Mario.
    Step 10 is where I’m stuck. Can you post your full logstash config please?

    I’ve tried something like this but still not seeing anything.

    input { tcp { port => 3514 }
    }
    output {
    elasticsearch { hosts => [“localhost:9200”] }
    stdout { codec => rubydebug }
    }

    • Hey Peter,

      I took a look at Sexilog afterwards but never got to write about it. Very nice solution to some problems but, at the time, I wasn’t aware of it. I wish I was still fiddling with ELK thou… Cheers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s