Apache Zookeeper Notes

Russell Bateman
November 2019

Main page: Welcome to Apache ZooKeeper™. Note the awkward supposition that "zookeeper" is two words in English; it isn't. For this reason, I'll always refer in here to the software as Zookeeper rather than ZooKeeper.

What is Zookeeper?

It's a centralized, clusterable service for maintaining configuration information, naming, providing distributive synchronization and providing group services. Kafka is a notable consumer of Zookeeper, probably to support its clustering, also NiFi absolutely for its clustering.

Zookeeper's claim is that clusters are hard to implement and applications will skimp on their implementation making them brittle, difficult to manage and prone to break.


Quickstart

These notes are far more complete and much clearer than any I found anywhere else on line.

  1. The download is (overly) complicated as compared to most downloads. You must dig around for a link to a mirror site, then navigate down through a filesystem to grab a tarball. I went here: Zookeeper Releases, then downloaded the binaries as apache-zookeeper-3.5.6-bin.tar.gz from zookeeper-3.5.6.

    Note: apache-zookeeper-3.5.6.tar.gz, only 3Mb against the 8Mb of the other tarball, contains only source code which you must build for yourself. You want the binaries instead.

  2. Copy the tarball, as root, to /opt and explode it. 
    ~/Downloads/zookeeper $ sudo bash
zookeeper # cp apache-zookeeper-3.5.6-bin.tar.gz /opt
zookeeper # cd /opt
/opt # tar -zxf apache-zookeeper-3.5.6-bin.tar.gz
  3. Create a subdirectory for Zookeeper state: 
    /opt # mkdir /var/lib/zookeeper
  4. Create the file /opt/apache-zookeeper-3.5.6-bin/conf/zoo.cfg to contain the following: 
    tickTime = 2000
dataDir = /var/lib/zookeeper
clientPort = 2181      # (ensure this port is available on your host1)
initLimit = 5
syncLimit = 2
    Note: We're not defining any other servers in our configuration (via server.n = ...) because, for this quick-start, we do not intend to use other than our local host.

  5. Start Zookeeper: 
    # /opt/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Starting zookeeper ... FAILED TO START
    Check the log to see why Zookeeper failed to start: 
    # /opt/apache-zookeeper-3.5.6-bin/logs/zookeeper-root-server-hostname.out
2019-11-07 08:40:17,460 [myid:] - INFO  [main:QuorumPeerConfig@133] - Reading configuration from:
/opt/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
2019-11-07 08:40:17,463 [myid:] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2019-11-07 08:40:17,464 [myid:] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
2019-11-07 08:40:17,468 [myid:] - ERROR [main:QuorumPeerMain@89] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /opt/apache-zookeeper-3.5...
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:113)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.lang.IllegalArgumentException: myid file is missing
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.checkValidity(QuorumPeerConfig.java:734)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:605)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:420)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
        ... 2 more
Invalid config, exiting abnormally
  6. This is because we did not create the file, /var/lib/zookeeper/myid. It should contain (nothing other than) "1".2 If we had defined other servers in our configuration, each one of them would need "2", "3",2 etc. in their myid file as well.

  7. Start Zookeeper again: 
    # /opt/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Starting zookeeper ... FAILED TO START
    Check the log to see why Zookeeper failed to start: 
    # /opt/apache-zookeeper-3.5.6-bin/logs/zookeeper-root-server-hostname.out
2019-11-07 08:47:05,956 [myid:] - INFO  [main:QuorumPeerConfig@133] - Reading configuration from: /opt/apache-z...
2019-11-07 08:47:05,959 [myid:] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2019-11-07 08:47:05,960 [myid:] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
2019-11-07 08:47:05,962 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
.
.
.
2019-11-07 08:47:06,076 [myid:1] - INFO  [main:ContextHandler@855] - Started o.e.j.s.ServletContextHandler@c8c1...
2019-11-07 08:47:06,079 [myid:1] - ERROR [main:ZooKeeperServerMain@79] - Unable to start AdminServer, exiting a...
org.apache.zookeeper.server.admin.AdminServer$AdminServerException: Problem starting AdminServer on address 0.0...
        at org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:107)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:138)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:8080
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.server.Server.doStart(Server.java:396)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:103)
        ... 5 more
Caused by: java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:461)
        at java.base/sun.nio.ch.Net.bind(Net.java:453)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
        ... 12 more
Unable to start AdminServer, exiting abnormally
    Zookeeper, like many Java-based servers, wants to use port 8080 by default (and I'm already using it for something else). Let's change that in /opt/apache-zookeeper-3.5.6-bin/conf/zoo.cfg: 
    tickTime = 2000
dataDir = /var/lib/zookeeper
clientPort = 2181
initLimit = 5
syncLimit = 2
admin.serverPort=18080 # (ensure this port is available on your host1)
  8. Try a third time: 
    # /opt/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.5.6-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

1 If this returns nothing, you can expect to be able to use port 2181: 

# lsof -i:2181

2  The double quotes are not part of what's in myid.