Ri Xu Online

Logstash

Logstash is a tool for receiving, processing and outputting logs. All kinds of logs. System logs, webserver logs, error logs, application logs, and just about anything you can throw at it. Sounds great, eh?

Logstash provides a powerful pipeline for storing, querying, and analyzing your logs. When using Elasticsearch as a backend data store and Kibana as a front-end reporting tool, Logstash acts as the workhorse. It includes an arsenal of built-in inputs, filters, codecs, and outputs, enabling you to harness some powerful functionality with a small amount of effort.

Elasticsearch

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. Elasticsearch is the second most popular enterprise search engine. It can be used to search all kinds of documents. It provides scalable search, has near real-time search, and supports multitenancy.

Kibana

Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. You use Kibana to search, view, and interact with data stored in Elasticsearch indices. You can easily perform advanced data analysis and visualize your data in a variety of charts, tables, and maps.

Kibana makes it easy to understand large volumes of data. Its simple, browser-based interface enables you to quickly create and share dynamic dashboards that display changes to Elasticsearch queries in real time.

I have install ELK, the main components - Elasticsearch 1.7.0, Logstash 1.5.4 and and Kibana 4.1.1 on Ubuntu 14.04 LTS Server Edition. Storage Nginx access log into Elasticsearch with Logstash and setup browser-based dynamic interface with Kibana.

Install Java 8

Elasticsearch and Logstash require Java, so we will install that now. We will install Oracle Java 8 (update 20 or later) because that is what Elasticsearch recommends. It should, however, work fine with OpenJDK, if we decide to go that route.

• Method 1

Add the Oracle Java PPA to apt:

$ sudo add-apt-repository -y ppa:webupd8team/java

Update my apt package database:

$ sudo apt-get update

Install the latest stable version of Oracle Java 8 with this command (and accept the license agreement that pops up):

$ sudo apt-get -y install oracle-java8-installer

Now that Java 8 is installed, let's install ElasticSearch.

• Method 2

Download Java SE from Oracle official website

I have download jdk-8u60-linux-x64.tar.gz. Create directory

$ sudo mkdir /usr/lib/jvm

Unzip to install directory

$ sudo tar -zxvf jdk-8* -C /usr/lib/jvm

Setup environment variable

$ sudo vim ~/.bashrc

Append following code

# Set Oracle JDK environment
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_60
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

Setup default JDK version of system

$ sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.8.0_60/bin/java 300
$ sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.8.0_60/bin/javac 300
$ sudo update-alternatives --install /usr/bin/jar jar /usr/lib/jvm/jdk1.8.0_60/bin/jar 300
$ sudo update-alternatives --install /usr/bin/javah javah /usr/lib/jvm/jdk1.8.0_60/bin/javah 300
$ sudo update-alternatives --install /usr/bin/javap javap /usr/lib/jvm/jdk1.8.0_60/bin/javap 300
$ sudo update-alternatives --config java

Test installation

$ java -version

Install Elasticsearch

Install Elasticsearch please reference the post Elasticsearch Cluster on Ubuntu

After install edit the configuration

$ sudo vim /etc/elasticsearch/elasticsearch.yml

To restrict outside access to my Elasticsearch instance (port 9200), so outsiders can't read my data or shutdown my Elasticsearch cluster through the HTTP API. Find the line that specifies network.host, uncomment it, and replace its value with IP address so it looks like this:

network.host: 192.168.171.129

Save and exit elasticsearch.yml.

Now restart Elasticsearch:

$ sudo service elasticsearch restart

Then run the following command to start Elasticsearch on boot up:

$ sudo update-rc.d elasticsearch defaults 95 10

Now that Elasticsearch is up and running, let's install Kibana.

Install Kibana

Download Kibana 4 to our home directory with the following command:

$ cd ~; wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

Extract Kibana archive with tar:

$ tar xvf kibana-*.tar.gz

Open the Kibana configuration file for editing:

$ vim ~/kibana-4*/config/kibana.yml

In the Kibana configuration file, find the line that specifies host, and replace the IP address (0.0.0.0 by default) with real IP address:

host: "192.168.171.129"

Save and exit. This setting makes it so Kibana will only be accessible to the IP address. This is fine because we will use an Nginx reverse proxy to allow external access.

Let's copy the Kibana files to a more appropriate location. Create the /opt directory with the following command:

$ sudo mkdir -p /opt/kibana

Now copy the Kibana files into my newly-created directory:

$ sudo cp -R ~/kibana-4*/* /opt/kibana/

Kibana can be started by running /opt/kibana/bin/kibana, but we want it to run as a service. Download a Kibana init script with this command:

$ cd /etc/init.d && sudo wget https://raw.githubusercontent.com/akabdog/scripts/master/kibana4_init && sudo mv kibana4_init kibana4

Now enable the Kibana service, and start it:

$ sudo chmod +x /etc/init.d/kibana4
$ sudo update-rc.d kibana4 defaults 96 9
$ sudo service kibana4 start

Before we can use the Kibana web interface, we have to set up a reverse proxy. Let's do that now, with Nginx.

Use apt to install Nginx

$ sudo apt-get install nginx

Now open the Nginx default server block in my favorite editor.

$ sudo vim /etc/nginx/sites-available/default

Delete the file's contents, and paste the following code block into the file. Be sure to update the server_name to match my server's name:

server {
    listen 80;

    server_name example.com;
    access_log  /var/log/nginx/kibana.myhost.org.access.log;

}

Install Logstash

Create the Logstash source list

$ echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list

Update our apt package database:

$ sudo apt-get update

Install Logstash with this command:

$ sudo apt-get install logstash

Logstash is installed but it is not configured yet.

Configure Logstash

Logstash configuration files are in the JSON-format, and reside in /etc/logstash/conf.d. The configuration consists of three sections: inputs, filters, and outputs.

Let's create a configuration file called logstash-sample.conf

$ sudo vim /etc/logstash/conf.d/logstash-sample.conf

Insert the following input configuration

input {
    file {
        #type => "nginx_access"
        codec => "json"
        path  => ["/var/log/nginx/kibana.myhost.org.access.log"]
    }
}
output {
    elasticsearch {
        embedded => false
        protocol => "http"
        host     => "192.168.171.129"
        port     => 9200
    }
}

Now, start Logstash to put our configuration changes into effect:

$ nohup /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-sample.conf &

In a web browser, visit public IP address on 5601 port (http://192.168.171.129:5601)

Go ahead and select @timestamp from the dropdown menu, then click the Create button to create the first index.

Now click the Discover link in the top navigation bar. By default, this will show you all of the log data over the last 15 minutes. We should see a histogram with log events, with log messages below:

Elasticsearch Browser

Now that my Nginx access logs are centralized via Elasticsearch and Logstash, and I'm able to visualize them with Kibana 4.

Reference Article How To Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04