Apache NiFi and Docker notes, part 2

Russell Bateman
March 2023
last update:



Build our own Dockerfile to insert our NAR

...removing single-user authentication. This returns the installation of Apache NiFi to a simple undertaking as existed prior to 1.14.0 when they succumbed to the perception that the product wasn't sophisticated enough out of the box.

The utility of this is to make life easier during the design, implementation, testing and debugging of custom NiFi processors. See How to get NiFi to work (unsecurely) as before.

Obtain version number (as associated with) tags from here.

For our notes on this page, we're going to build two different containers:


Create an Apache NiFi 1.13.2 Dockerfile and insert our NAR

Starting in our Docker-container development directory... 

$ ll
total 222852
drwxrwxr-x 2 russ russ      4096 Mar 16 12:50 .
drwxrwxr-x 4 russ russ      4096 Mar 16 12:50 ..
-rw-rw-r-- 1 russ russ       246 Mar 16 12:49 Dockerfile
-rw-rw-r-- 1 russ russ 228186423 Mar 10 12:17 legacy-pipeline.nar

FROM    apache/nifi:1.13.2
LABEL   maintainer="[email protected]"
USER    nifi
ADD     legacy-pipeline.nar /opt/nifi/nifi-current/extensions
EXPOSE  8080
ENV     PATH=/opt/nifi/nifi-current/bin:$PATH
CMD     [ "nifi.sh", "start" ]

Let's watch this image build... 

$ docker build -t legacy-nifi .
Sending build context to Docker daemon  228.2MB
Step 1/7 : FROM    apache/nifi:1.13.2
1.13.2: Pulling from apache/nifi
e22122b926a1: Pull complete
f29e09ae8373: Pull complete
e319e3daef68: Pull complete
3c6c60bfcb21: Pull complete
a5ee52e88d8b: Pull complete
7fd26b251c91: Pull complete
680bf38669f4: Pull complete
b477f3e85bd2: Pull complete
0dd0aa8a0dda: Pull complete
3c02cc8bf5ba: Pull complete
f2fd7d72dbbc: Pull complete
888dd3524c2a: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:2f4dbbe29ec83d8f43ef36ab07c1e4a139e5c2f75bab538c2bf7ee53d1b93bd7
Status: Downloaded newer image for apache/nifi:1.13.2
 ---> 6ca96dc25d43
Step 2/7 : LABEL   maintainer="[email protected]"
 ---> Running in 2cfeae84d886
Removing intermediate container 2cfeae84d886
 ---> 6b032977f919
Step 3/7 : USER    nifi
 ---> Running in 6bbe1259407f
Removing intermediate container 6bbe1259407f
 ---> a83527674ca8
Step 4/7 : ADD     legacy-pipeline.nar /opt/nifi/nifi-1.13.2/extensions
 ---> 3d83c42ce4bf
Step 5/7 : EXPOSE  8080
 ---> Running in d0b256ae3e44
Removing intermediate container d0b256ae3e44
 ---> 66974643afb2
Step 6/7 : ENV     PATH=/opt/nifi/nifi-1.13.2/bin:$PATH
 ---> Running in 10fcb0afcc98
Removing intermediate container 10fcb0afcc98
 ---> 35a7a80831e2
Step 7/7 : CMD     [ "nifi.sh", "start" ]
 ---> Running in b01cf9602922
Removing intermediate container b01cf9602922
 ---> 85a0f0375ff3
Successfully built 85a0f0375ff3
Successfully tagged legacy-nifi:latest
$ docker images
REPOSITORY     TAG                  IMAGE ID       CREATED         SIZE
legacy-nifi    latest               85a0f0375ff3   4 minutes ago   2.25GB
apache/nifi    1.13.2               6ca96dc25d43   24 months ago   2.03GB
$ docker run -d --name legacy-nifi -p 2222:8080 legacy-nifi:latest
b1ad2fb193ac50b51cf1d1e10d691616f52fc265fb28da9952d5ef8dbc92c8e5
$ docker ps -a
CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS                                                                      NAMES
b1ad2fb193ac   legacy-nifi:latest   "../scripts/start.sh…"   39 seconds ago   Up 38 seconds   8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:2222->8080/tcp, :::2222->8080/tcp   legacy-nifi
$ docker exec -it legacy-nifi bash
nifi@b1ad2fb193ac:/opt/nifi$ alias ll='ls -alg'
nifi@b1ad2fb193ac:/opt/nifi$ ll
total 28
drwxr-xr-x 1 nifi 4096 Mar 19  2021 .
drwxr-xr-x 1 root 4096 Mar 16  2021 ..
lrwxrwxrwx 1 nifi   22 Mar 19  2021 nifi-1.13.2 -> /opt/nifi/nifi-current
drwxr-xr-x 1 nifi 4096 Mar 16 20:42 nifi-current
lrwxrwxrwx 1 nifi   30 Mar 19  2021 nifi-toolkit-1.13.2 -> /opt/nifi/nifi-toolkit-current
drwxr-xr-x 6 nifi 4096 Jan 22  2020 nifi-toolkit-current
drwxr-xr-x 1 nifi 4096 Mar 16  2021 scripts
nifi@b1ad2fb193ac:/opt/nifi$ ll nifi-current
total 356
drwxr-xr-x    1 nifi   4096 Mar 16 20:42 .
drwxr-xr-x    1 nifi   4096 Mar 19  2021 ..
-rw-r--r--    1 nifi 171463 Jan 22  2020 LICENSE
-rw-r--r--    1 nifi  92157 Jan 22  2020 NOTICE
-rw-r--r--    1 nifi   4784 Jan 22  2020 README
drwxr-xr-x    1 nifi   4096 Jan 22  2020 bin
drwxr-xr-x    3 nifi   4096 Mar 16 20:42 conf
drwxr-xr-x 1026 nifi  20480 Mar 16 20:42 content_repository
drwxr-xr-x    2 nifi   4096 Mar 16 20:42 database_repository
drwxr-xr-x    3 nifi   4096 Jan 22  2020 docs
drwxr-xr-x    1 nifi   4096 Mar 16 20:36 extensions
drwxr-xr-x    3 nifi   4096 Mar 16 20:42 flowfile_repository
drwxrwx---    5 nifi  12288 Jan 22  2020 lib
drwxr-xr-x    2 nifi   4096 Mar 16 20:42 logs
drwxr-xr-x    2 nifi   4096 Mar 19  2021 provenance_repository
drwxr-xr-x    2 nifi   4096 Mar 16 20:42 run
drwxr-xr-x    3 nifi   4096 Mar 16 20:42 state
drwxr-xr-x    5 nifi   4096 Mar 16 20:42 work
nifi@b1ad2fb193ac:/opt/nifi$ ll nifi-current/extensions/
total 222852
drwxr-xr-x 1 nifi      4096 Mar 16 20:36 .
drwxr-xr-x 1 nifi      4096 Mar 16 20:42 ..
-rw-rw-r-- 1 root 228186423 Mar 10 19:17 legacy-pipeline.nar

...and, of course, we have no problem getting into NiFi via http://localhost:2222/nifi where we can place one of legacy-pipeline.nar's processors on its canvas. And NiFi's shipping, standard processors can be placed in the flow as well. This is how we define success.


Create an Apache NiFi 1.19.1 Dockerfile and insert our NAR

Moving forward, 1.19.1 will be no harder to erect that was the one we created to run legacy-pipeline.nar, but we're going to use our modern (non-legacy) NAR. 

FROM    apache/nifi:1.19.1
LABEL   maintainer="[email protected]"
USER    nifi
ADD     modern-pipeline.nar /opt/nifi/nifi-current/extensions
EXPOSE  8080
ENV     PATH=/opt/nifi/nifi-current/bin:$PATH
CMD     [ "nifi.sh", "start" ]

Useful links




Dockerfile notes

  1. FROM —pulls (includes) dependencies from DockerHub image configurations.

  2. LABEL —who the maintainer/guilty party is.

  3. USER —the system user that should be doing the work.

  4. RUN —is used to perform shell commands in the container, for example, mkdir and sed, as if the container were a running VM and you were making changes to its filesystem, etc. Each RUN command creates a new layer in the image (which is suboptimal); chain your RUN commands together whenever possible. However, we don't need a RUN command this time.

  5. ADD —is used to copy files or subdirectories into a Docker image. ADD has the effect of adding a new layer (image) to the existing image.

  6. EXPOSE —expose the listed port(s) for mapping to the host via the -p option when launching a container. This is more documentary than effectual, but we'll add it. It means basically that port 8080 inside our container will be the one Apache NiFi listens on. The way this really happens though is via the -p option on 
    $ docker run

  7. ENV —update the container's PATH environment variable in order that CMD may find the command it's going to execute once the container is running.

  8. CMD —used to run NiFi in the image, along with any arguments. Allows you to set a default command plus options and arguments that will be executed only when the container is run. However, if the container is run with a command on the command line, this command is ignored.