Apache Kafka is a distributed streaming platform. A Kafka cluster is not only highly scalable and fault-tolerant, but it also has a much higher throughput compared to other message brokers such as ActiveMQ and RabbitMQ. Though it is generally used as a pub/sub messaging system, a lot of organizations also use it for log aggregation because it offers persistent storage for published messages.
Ubuntu Server 17.10 Artful Aardvark 4GB RAM Scala 2.11 Kafka 1.0.0
Before installing additional packages, update the list of available packages so you are installing the latest versions available in the repository:
$ sudo apt-get update
As Apache Kafka needs a Java runtime environment, use
apt-get to install the
$ sudo apt-get install default-jre $ java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.17.10.2-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
Create a user called
kafka using the
$ sudo useradd kafka -m
Set its password using passwd:
$ sudo passwd kafka
Add it to the
sudo group so that it has the privileges required to install Kafka's dependencies. This can be done using the
$ sudo adduser kafka sudo
The Kafka user is now ready. Log into it using
$ su - kafka
Apache ZooKeeper is an open source service built to coordinate and synchronize configuration information of nodes that belong to a distributed system. A Kafka cluster depends on ZooKeeper to perform—among other things—operations such as detecting failed nodes and electing leaders.
Since the ZooKeeper package is available in Ubuntu's default repositories, install it using
$ sudo apt-get install zookeeperd
After the installation completes, ZooKeeper will be started as a daemon automatically. By default, it will listen on port
To make sure that it is working, connect to it via Telnet:
$ telnet localhost 2181
At the Telnet prompt, type in
ruok and press ENTER.
If everything's fine, ZooKeeper will say
imok and end the Telnet session.
$ echo ruok | nc localhost 2181
Download and Extract Kafka Binaries
Now that Java and ZooKeeper are installed, it is time to download and extract Kafka.
To start, create a directory called
Downloads to store all your downloads.
$ mkdir -p ~/Downloads
wget to download the Kafka binaries.
$ wget "http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz" -O ~/Downloads/kafka.tgz
Create a directory called
kafka and change to this directory. This will be the base directory of the Kafka installation.
$ mkdir -p ~/kafka && cd ~/kafka
Extract the archive you downloaded using the tar command.
$ tar -xvzf ~/Downloads/kafka.tgz --strip 1
Configure the Kafka Server
The next step is to configure the Kakfa server, open
server.properties using text editor:
$ vim ~/kafka/config/server.properties
By default, Kafka doesn't allow you to delete topics. To be able to delete topics, add the following line at the end of the file:
Save the file and exit the text editor.
Start the Kafka Server
kafka-server-start.sh script using
nohup to start the Kafka server (also called Kafka broker) as a background process that is independent of your shell session.
$ ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties
Run service in daemon:
$ nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &
Wait for a few seconds for it to start. You can be sure that the server has started successfully when you see the following messages in
[2017-12-09 17:16:29,521] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
You now have a Kafka server which is listening on port
Test the Installation
Let us now create a topic named
test with a single partition and only one replica:
$ ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
We can now see that topic if we run the list topic command:
$ ~/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181
Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.
Send some messages
Run the producer and then type a few messages into the console to send to the server.
$ ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >This is a message >This is another message
Kafka also has a command line consumer that will dump out messages to standard output.
$ ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning >This is a message >This is another message
Restrict the Kafka User
Now that all installations are done, you can remove the
kafka user's admin privileges. Before you do so, log out and log back in as any other non-root
sudo user. If you are still running the same shell session you started the installation, simply type
To remove the Kafka user's admin privileges, remove it from the
$ sudo deluser kafka sudo
To further improve your Kafka server's security, lock the
kafka user's password using the
passwd command. This makes sure that nobody can directly log into it.
$ sudo passwd kafka -l
At this point, only root or a sudo user can log in as
kafka by typing in the following command:
$ sudo su - kafka
In the future, if you want to unlock it, use passwd with the
$ sudo passwd kafka -u
You now have a secure Apache Kafka running on your Ubuntu server. You can easily make use of it in your projects by creating Kafka producers and consumers using Kafka clients which are available for most programming languages. To learn more about Kafka, do go through its documentation.