Install Apache Kafka on Ubuntu

Wiki

Apache Kafka is a distributed streaming platform. A Kafka cluster is not only highly scalable and fault-tolerant, but it also has a much higher throughput compared to other message brokers such as ActiveMQ and RabbitMQ. Though it is generally used as a pub/sub messaging system, a lot of organizations also use it for log aggregation because it offers persistent storage for published messages.

Environment

Ubuntu Server 17.10 Artful Aardvark 4GB RAM
Scala 2.11 Kafka 1.0.0

Install Java

Before installing additional packages, update the list of available packages so you are installing the latest versions available in the repository:

$ sudo apt-get update

As Apache Kafka needs a Java runtime environment, use apt-get to install the default-jre package:

$ sudo apt-get install default-jre
$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.17.10.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

Create a user called kafka using the useradd command:

$ sudo useradd kafka -m

Set its password using passwd:

$ sudo passwd kafka

Add it to the sudo group so that it has the privileges required to install Kafka's dependencies. This can be done using the adduser command:

$ sudo adduser kafka sudo

The Kafka user is now ready. Log into it using su:

$ su - kafka

Install ZooKeeper

Apache ZooKeeper is an open source service built to coordinate and synchronize configuration information of nodes that belong to a distributed system. A Kafka cluster depends on ZooKeeper to perform—among other things—operations such as detecting failed nodes and electing leaders.

Since the ZooKeeper package is available in Ubuntu's default repositories, install it using apt-get.

$ sudo apt-get install zookeeperd

After the installation completes, ZooKeeper will be started as a daemon automatically. By default, it will listen on port 2181.

To make sure that it is working, connect to it via Telnet:

$ telnet localhost 2181

At the Telnet prompt, type in ruok and press ENTER.

If everything's fine, ZooKeeper will say imok and end the Telnet session.

$ echo ruok | nc localhost 2181

Download and Extract Kafka Binaries

Now that Java and ZooKeeper are installed, it is time to download and extract Kafka.

To start, create a directory called Downloads to store all your downloads.

$ mkdir -p ~/Downloads

Use wget to download the Kafka binaries.

$ wget "http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz" -O ~/Downloads/kafka.tgz

Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation.

$ mkdir -p ~/kafka && cd ~/kafka

Extract the archive you downloaded using the tar command.

$ tar -xvzf ~/Downloads/kafka.tgz --strip 1

Configure the Kafka Server

The next step is to configure the Kakfa server, open server.properties using text editor:

$ vim ~/kafka/config/server.properties

By default, Kafka doesn't allow you to delete topics. To be able to delete topics, add the following line at the end of the file:

delete.topic.enable=true

Save the file and exit the text editor.

Start the Kafka Server

Run the kafka-server-start.sh script using nohup to start the Kafka server (also called Kafka broker) as a background process that is independent of your shell session.

$ ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

Run service in daemon:

$ nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &

Wait for a few seconds for it to start. You can be sure that the server has started successfully when you see the following messages in ~/kafka/kafka.log:

[2017-12-09 17:16:29,521] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

You now have a Kafka server which is listening on port 9092.

Test the Installation

Let us now create a topic named test with a single partition and only one replica:

$ ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

We can now see that topic if we run the list topic command:

$ ~/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181

Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.

Send some messages

Run the producer and then type a few messages into the console to send to the server.

$ ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>This is a message
>This is another message

Kafka also has a command line consumer that will dump out messages to standard output.

$ ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
>This is a message
>This is another message

Restrict the Kafka User

Now that all installations are done, you can remove the kafka user's admin privileges. Before you do so, log out and log back in as any other non-root sudo user. If you are still running the same shell session you started the installation, simply type exit.

To remove the Kafka user's admin privileges, remove it from the sudo group.

$ sudo deluser kafka sudo

To further improve your Kafka server's security, lock the kafka user's password using the passwd command. This makes sure that nobody can directly log into it.

$ sudo passwd kafka -l

At this point, only root or a sudo user can log in as kafka by typing in the following command:

$ sudo su - kafka

In the future, if you want to unlock it, use passwd with the -u option:

$ sudo passwd kafka -u

Conclusion

You now have a secure Apache Kafka running on your Ubuntu server. You can easily make use of it in your projects by creating Kafka producers and consumers using Kafka clients which are available for most programming languages. To learn more about Kafka, do go through its documentation.

Install Apache Kafka on Ubuntu
3 votes, 5.00 avg. rating (98% score)