The ELK stack powered by Docker – Updated !

Hola,

In a previous post, I’ve introduced the ELK stack powered by Docker & Fig (see the ELK stack powered by Docker).

I’ve recently decided to update the project to replace the usage of fig with compose and to replace all my custom images with the latest official images !

It is now based on the following Docker images available on Dockerhub:

01/11/2015 : Project updated !

As the project is based on the latest Docker images versions, it means Elasticsearch 2.x, Logstash 2.x and Kibana 4.2.x ! Feel free to discover the new features of these releases (have a look here: https://www.elastic.co/blog/release-we-have).

Note: For the nostalgic folks, you can still access the 1.x version (Elasticsearch 1.x, Logstash 1.x and Kibana 4.1.x) on the 1.x branch ! Here it is: https://github.com/deviantony/docker-elk/tree/1.x

Usage

Pre-requisites

You’ll need Docker and Docker Compose.

The following installation procedures have been tested on Ubuntu 14.04.

Docker installation

Use the following command to install Docker:

$ curl -sSL https://get.docker.com/ubuntu/ | sudo sh

Docker Compose installation

Follow the procedure available at https://docs.docker.com/compose/install/ to install the latest version of Docker Compose.

Use the stack

First, you’ll need to checkout the git repository:

$ git clone https://github.com/deviantony/docker-elk.git

By default, the stack is shipped with a simple Logstash configuration, it will listen for any TCP input on port 5000.

Then start the stack using Compose:

$ cd docker-elk
$ docker-compose up

Compose will start a container for each service of the ELK stack and output their logs.

If you’re still using the default input configuration for Logstash, you can inject some data into Elasticsearch from a file:

$ nc localhost 5000 < /some/log/file.log

Then you can check the results in Kibana by hitting the following URL in your browser: http://localhost:5601

Enjoy 🙂

The ELK stack powered by Docker

UPDATE: The stack is now powered by docker-compose and using the latest official images for Elasticsearch/Logstash/Kibana. See my new article https://deviantony.wordpress.com/2015/07/23/the-elk-stack-powered-by-docker-updated/ — 23/07/2015

Ahoy,

I’ve recently created a solution to setup an ELK stack using the Docker engine, it can be used to:

  • Quickly boot an ELK stack for demo purposes
  • Use it to test your Logstash configurations and see the imported data in Elasticsearch
  • Import a subset of data into Elasticsearch and prepare dashboards on Kibana 3
  • Use Kibana 4 !

UPDATE: The stack is now fully functionnal with docker-compose as a replacement for fig. See http://docs.docker.com/compose/install/ for docker-compose installation. — 27/02/2015

The solution is available on Github: https://github.com/deviantony/docker-elk

It is based on multiple Docker images available on Dockerhub:

  • elk-elasticsearch: Latest 1.5 stable version of Elasticsearch + Marvel with Oracle Java JDK 7
  • elk-logstash: Latest 1.4 stable version of Logstash with Oracle Java JDK 7
  • elk-kibana: Kibana 3.1.2 or Kibana 4

Prerequisites

You’ll need Docker and Fig.

The following installation procedures have been tested on Ubuntu 14.04.

Docker installation

Use the following command to install Docker:

$ curl -sSL https://get.docker.com/ubuntu/ | sudo sh

Fig installation

Fig is available via pip, a tool dedicated to manage Python packages.

First, you’ll need to install pip if it is not already present on your system:

$ sudo apt-get install python-pip

Then, install fig:

$ sudo pip install -U fig

Use the stack

First, you’ll need to checkout the git repository:

$ git clone https://github.com/deviantony/fig-elk.git

By default, the stack is shipped with a simple Logstash configuration, it will listen for any TCP input on port 5000.

You can update the Logstash configuration by updating the file logstash-conf/logstash.conf (to test your filters, for example).

Then start the stack using fig:

$ cd fig-elk
$ fig up

Fig will start a container for each service of the ELK stack and output their logs.

If you’re still using the default input configuration for Logstash, you can inject some data into Elasticsearch from a file:

$ nc localhost 5000 < /some/log/file.log

Then you can check the results in Kibana 3, by hitting the following URL in your browser: http://localhost:8080

Or, if you’d like to use Kibana 4, hit the following URL: http://localhost:5601

Also Elasticsearch is shipped with Marvel, you have access to the cluster monitoring on the following URL http://localhost:9200/_plugin/marvel

Have fun with the ELK stack 🙂

VMWare tips

Here is a few tips I’ve learned with VMWare virtual machines hosting Ubuntu 12.04 / 14.04.

How to refresh an upgraded disk

If you want to resize a disk at runtime, go to your VM preferences and update your disk size. Now, how to refresh the state of the disk in the virtual machine without rebooting it?

For each disk in your VM, there is a related directory in the /sys/class/scsi_devices directory. Use the following command to rescan your disk capacity:

$ echo 1 > /sys/class/scsi_device/DISK/device/rescan

How to refresh new disk

If you want to add another disk at runtime, go to your VM preferences and add a new disk. Now, how to refresh the system so it can see the new disk without rebooting the VM?

Basically, you just need to rescan the SCSI bus to which the storages are connected.

The following commands assume you have root access on the system.

Find your host bus number

$ grep mpt /sys/class/scsi_host/host?/proc_name

This will return a line like the following:

/sys/class/scsi_host/host0/proc_name:mptspi

Where host0 is the relevant bus number.

Rescan the SCSI bus

$ echo "- - -" > /sys/class/scsi_host/host0/scan