Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu

In this lesson, we will see how we can get our ELK Stack up and running on our Ubuntu machines. ELK consist of Elasticsearch 2.2.x, Logstash 2.2.x, and Kibana 4.5.x. We will show how we can configure this stack and use Kibana to visualise the logs which our applications and systems create in a centralized location, using Filebeat 1.1.x. To understand these tools, let’s look at a basic definition for each of them:

• Logstash: It is an Open Source tool which is used to collect, parse and store logs for an application or a system which can be used later for metrics of APIs, errors faced in a system and many more use-cases which are beyond the scope of this lesson
• Kibana: This is a dashboard interface on the web which is an excellent dashboard used to search and view the logs that Logstash has indexed into the Elasticsearch index
• Filebeat: This is installed on the client-server who want to send their logs to Logstash. Filebeat acts as a log shipping agent and communicates with Logstash.

Let us also visualise how things will work:

Prerequisites

For this lesson and all installations it needs, you should have root access to the machine. We will be using a machine with this configuration:

• Ubuntu 16.04
• RAM: 4 GB
• CPU: 2

Few application servers from where you want to gather data from would also be a good to have.

Install Java

To install Elasticsearch on Ubuntu, we must install Java first. Java might not be installed by default. We can verify it by using this command:

Checking Java version

Here is what we get back with this command:

We will now install Java on our system. Use this command to do so:

Installing Java

Once these commands are done running, we can again verify that Java is now installed by using the same version command.

Install Elasticsearch

Next step for the ELK Stack setup is installing Elasticsearch on Ubuntu Machine which will store the logs generated by systems and applications. Before we can install Elasticsearch, we need to import its public GPG keys to the rpm package manager:

GPG Keys

Now, insert the mentioned lines to the configuration file for the repository ‘elasticsearch.repo’:
Repository Config

Now, read the lesson Install ElasticSearch on Ubuntu for installation process. Once ES is up and running, make sure it responds normally to this curl command:

ES Status

The normal output will be:

Install Logstash

Installing Logstash is very easy using the apt package manager and is available with the same repository and public key as Elasticsearch, so we don’t have to do that again. Let’s create the source list to start:

Create Source list

Update the apt package list:

Updating Packages

Install Logstash with a single command:

Install Logstash

Logstash is installed but it is not configured yet. We will configure Logstash in coming sections.

Install Kibana

Kibana is very easy to install. We can start by creating the Kibana source list:

Create Kibana source list

Now, we will update the apt package list:

Updating Packages

We are ready to install Kibana now:

Install Kibana

Once Kibana is installed, we can run it:

Start Kibana Service

Before we show you the Kibana Dashboard, we need to setup the Filebeat Log shipping agent as well.

Setup Filebeat

We are ready to install Filebeat now:

Install Filebeat

Before we can start the Filebeat service, we need to configure it for the input type and document type. Because we’re using system logs only as of now, let’s mention this in the configuration file in ‘/etc/filebeat/filebeat.yml’:

Configure Filebeat

We can also start filebeat now:

Start Filebeat Service

Once filebeat is up and running, we can check that it is OK by issuing the following curl command:

Testing Filebeat

We should receive a similar result as we got in the ES installation.

Connecting to Kibana

We are now ready to connect to Kibana. As we already started the Kibana service, its dashboard should be visible at:

Kibana Dashoboard URL

Once you’re up on Kibana, create an index on Kibana with name ‘filebeat-*’. Now based on the logs available, you can see the metrics and logs in your Kibana Dashboard:

Conclusion

In this lesson, we looked at how we can install and start using the ELK Stack for log visualisation and support an excellent Dashboard for business teams.