Apache NiFi Guided Tour, part 3:
Injecting a NiFi Custom-processor
into an Installed Instance of Apache NiFi

Russell Bateman
March 2023
last update:


Table of Contents

Guided Tour, part 1
Guided Tour, part 2

This is continued from Apache NiFi Guided Tour, part 2 in which we download a custom processor already written to bring it up in IntelliJ IDEA.

Build the NAR

Outside of IDEA, at the command line, build the NAR thus: 

~/dev/filter $ mvn clean package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] filter-parent                                                      [pom]
[INFO] processor                                                          [jar]
[INFO] filter                                                             [nar]
[INFO]
[INFO] ---------------< com.windofkeltia.filter:filter-parent >----------------
[INFO] Building filter-parent 1.0                                         [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ filter-parent ---
[INFO]
[INFO] -----------------< com.windofkeltia.filter:processor >------------------
[INFO] Building processor 1.0                                             [2/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] ...
[INFO]
[INFO] --- nifi-nar-maven-plugin:1.4.0:nar (default-nar) @ filter ---
[INFO] Copying nifi-api-1.19.1.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/nifi-api-1.19.1.jar
[INFO] Copying jakarta.activation-1.2.2.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/jakarta.activation-1.2.2.jar
[INFO] Copying processor-1.0.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/processor-1.0.jar
[INFO] Copying jaxb-runtime-2.3.5.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/jaxb-runtime-2.3.5.jar
[INFO] Copying jakarta.xml.bind-api-2.3.3.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/jakarta.xml.bind-api-2.3.3.jar
[INFO] Copying txw2-2.3.5.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/txw2-2.3.5.jar
[INFO] Copying nifi-utils-1.19.1.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/nifi-utils-1.19.1.jar
[INFO] Copying slf4j-api-2.0.6.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/slf4j-api-2.0.6.jar
[INFO] Copying istack-commons-runtime-3.0.12.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/istack-commons-runtime-3.0.12.jar
[INFO] Copying slf4j-simple-2.0.6.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/slf4j-simple-2.0.6.jar
[INFO] Copying jakarta.activation-api-1.2.2.jar to /home/russ/dev/filter/nar/target/classes/META-INF/bundled-dependencies/jakarta.activation-api-1.2.2.jar
[INFO] Generating documentation for NiFi extensions in the NAR...
[INFO] Found a dependency on version 1.19.1 of NiFi API
[INFO] Building jar: /home/russ/dev/filter/nar/target/filter-1.0.nar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for filter-parent 1.0:
[INFO]
[INFO] filter-parent ...................................... SUCCESS [  0.029 s]
[INFO] processor .......................................... SUCCESS [  1.596 s]
[INFO] filter ............................................. SUCCESS [  0.317 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.505 s
[INFO] Finished at: 2023-03-15T14:49:53-06:00
[INFO] ------------------------------------------------------------------------
~/dev/filter $ ll nar/target
total 1632
drwxrwxr-x 5 russ russ    4096 Mar 15 14:49 .
drwxrwxr-x 3 russ russ    4096 Mar 15 14:49 ..
drwxrwxr-x 3 russ russ    4096 Mar 15 14:49 classes
-rw-rw-r-- 1 russ russ 1648352 Mar 15 14:49 filter-1.0.nar
drwxrwxr-x 2 russ russ    4096 Mar 15 14:49 maven-archiver
drwxrwxr-x 3 russ russ    4096 Mar 15 14:49 META-INF

Inject the new NAR into NiFi

The NAR we'll inject is named filter-1.0.nar. Here's how:

In my environment, Apache NiFi is installed on the path, ~/dev/nifi/nifi-1.19.1. We call this NIFI_ROOT.

  1. Copy this file to the path ${NIFI_ROOT}/custom-lib.

  2. Bounce NiFi from the command line thus: 
    $ ${NIFI_ROOT}/bin/nifi.sh restart
  3. If you cannot seem to get NiFi started and need to watch errors and warning come out, you could do this: 
    $ cd ${NIFI_ROOT}                     # set working directory to NiFi's root
$ rm ./logs/*.log                     # remove existing logs to reduce clutter
$ ./bin/nifi.sh restart               # bounce NiFi
$ tail -f ./logs/nifi-app.log         # tail the log that will likely tell you what's wrong

The error most likely to keep NiFi from coming up (given that you've already experienced a working NiFi in part 1 of this guide) is that there's something with the NAR you built.

Use custom processor, Filter, in your flow

  1. In the toolbar at the top of the NiFi canvas, click (the first tool icon from the left) and drag a new processor down onto the canvas. A dialog will open. In the Filter edit field, type "Filter" (that we named our custom processor, "Filter," is a coincidence), then select that processor from the list of processors presented and click Add.

  2. Configure Filter by right-clicking on it, choosing Configure, then...
    1. Click the Properties tab; there will be no listed properties, so we'll create a dynamic property.
    2. Click the Add Property button (it's a large + sign at upper right).
    3. In the Add Property dialog, fill in the Property Name. Specially, the property name we choose will be used by Filter as a string to search on in the content of passing flowfiles. Let's choose "quick brown" as our property name.
    4. This isn't a sensitive value so we'll leave that setting as it came (i.e.: No). Click the OK button.
    5. In the next, tiny dialog, there is an edit field to hold the property value. We want Filter to replace all occurences of "quick brown" with "wily red" as the replacement string. Click OK.
    6. Having added a new property, click the Apply button to finish configuring this processor.

  3. Next, create a second success arc from GenerateFlowFile to Filter. This means that GenerateFlowFile will send copies of any flowfiles it creates to both the Wait processor and our custom processor, Filter. Hover over GenerateFlowFile, click the arrow that appears and drag to Filter.

  4. At this point, you should have begun to recognize a pattern. Processors produce successful output (and, often, failed output) and we use arcs (called relationships in NiFi) to connect successes and failures to other processors graphically.

  5. Now connect (in two click and drag actions) both success and failure from Filter to the Wait processor.

Run our flow

Let's try out our flow.

  1. Quickly start and stop GenerateFlowFile. This should fill the queue between GenerateFlowFile and Wait with a flowfile and also Filter with another (identical) flowfile. I see this:

  2. The reason we have parked a flowfile between GenerateFlowFile and Wait is so we can compare the original flowfile content with the changed content by Filter.

  3. Now start Filter then refresh the canvas (right-click in an empty place and choose Refresh). I see this:

  4. Notice that the size of the original flowfile (on its way to Wait) is 142 bytes while the output from Filter is only 139 bytes. If we examine the contents (how to do this was explained in part 1), we see that they are: 
    This is a test of the Emergency Broadcast System. This is only a test.
The wily red fox jumped over the lazy dog's back and got clean away.
    while in the original flowfile they are: 
    This is a test of the Emergency Broadcast System. This is only a test.
The quick brown fox jumped over the lazy dog's back and got clean away.

Here ends our Apache NiFi guided tour of installing NiFi, creating a custom processor and a flow that uses that new processor plus existing standard processors. For more information on Apache NiFi, consider the following links: