Logstash – Debug configuration

This post is a continuation of my previous post about the ELK stack setup, see here: how to setup an ELK stack.

I’ll show you how I’m using the logstash indexer component to start a debug process in order to test the logstash filters.

The aim is to start the indexer to parse the stdin so you can try inputs on the command line and see directly the result on stdout.

Update: I’ve recently created a tool to start a volatile ELK stack, you can also use it to test your filters: check it here.

Configuration

You’ll need to setup a configuration equivalent to the default one in /etc/logstash/conf.d, let’s say /etc/logstash/debug.d with the following files:

  • 00_input.conf
  • 02_filter_debug.conf
  • 10_output.conf

Input section

We are gonna told logstash to use stdin as it’s input.

/etc/logstash/conf.d/00_input.conf

input {
    stdin { }
}

Filter section

In the 02_filter_debug.conf file, you’ll define the filters you want to test.

filter {
 grok,mutate,drop...
}

Output section

The output will be stdout, so you can see the result (in JSON) of the filter processing directly in the console. We will also told logstash to duplicate the output into a file.

/etc/logstash/conf.d/10_output.conf

output {
    stdout {
        codec => "json"
    }
    file {
        codec => "json"
        path => "/tmp/debug-filters.json"
    }
}

Debugging

Now that the configuration is done, you’ll need to start the logstash binary with the debug configuration folder as a parameter:

$ /opt/logstash/bin/logstash -f /etc/logstash/debug.d -l /var/log/logstash/logstash-debug.log

The agent will take a few seconds to start, and then you’re ready for debug ! All you’ve got to do is copy your text in the command line and logstash will apply the filters defined in the filter section to it, then it will output the result on the command line.

To interrupt the logstash process, you’ll need to type the following commands: Ctrl+C and then Ctrl+D.

Have fun with your logs !

Advertisements

One thought on “Logstash – Debug configuration

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s