Notes on Remote Debugging in Apache Nifi Russell Bateman February 2025 last update:

(See an update to these instructions here, Tuesday, 28 March 2024.)

Steps to debugging a NiFi processor you've written. See this article to learn how remote debugging works in IntelliJ. The example uses IntelliJ to debug an application running in Tomcat, but it's a similar thing no matter if you're using Eclipse (or IntelliJ) to debug NiFi.

1. Edit conf/bootstrap.conf and simply uncomment the line that starts java.arg.debug. If you like, change the port number (address) to what you want to use in steps 2e or 2f below.
2. Create a Debug configuration.

   In IntelliJ IDEA, ...

   1. Do Run → Edit Configurations.
   2. Click + to add a new configuration.
   3. Choose Remote (because it's a remote debugging session you want to create).
   4. Give a Name:, something like "Local NiFi".
   5. Under Settings, change the Port: to 8000 or to whatever you established in conf/bootstrap.conf. (Note: this number has nothing to do with launching your browser with http://localhost:8080/nifi)
   6. If you're only debugging a single processor in a project with multiple modules, set the drop-down Search sources using module's classpath: to the module in which it lives.

   In Eclipse, do

   1. Do Run → Debug Configurations....
   2. Choose Remote Java Application.
   3. Click on the New Launch Configuration icon (a tiny sheet of paper with a yellow plus sign in upper left of dialog).
   4. Give it a name like "Local NiFi".
   5. In Project:, type the name (or browse for it) of the project containing your processor code.
   6. Set the Port: to 8000 or whatever you established in conf/bootstrap.conf. (Note: this number has nothing to do with launching your browser with http://localhost:8080/nifi)
   7. Click Apply or, if you're ready, Debug.

Note that you likely need to punch port 8000 (or whatever you used) through the remote host's firewall if there is one in place (and there likely is). You can do this a number of ways, but here's how to do it using iptables:

# iptables -A INPUT -p tcp --dport 8000 -j ACCEPT

If this is a secure server, you might not want to open such a port, but set up a tunnel instead. See the note on using a tunnel at the end of these steps.

3. Ensure that the NAR NiFi is using matches the code in which you plan to set breakpoints. (If it doesn't match, strange unworking symptoms may make you think that remote debugging does not work.)
4. Launch Apache NiFi (or bounce it).
5. Set one or more breakpoints in your processor code.
6. Launch the debugger (Run → Debug).

   In IntelliJ IDEA, do Run → Debug and choose "Local NiFi" (or whatever you called it) from the list presented. This brings up the debugger and displays, "Connected to the target VM, address 'localhost:8000', transport: 'socket'

   In Eclipse, do Run → Debug Configurations..., scroll down and choose "Local NiFi" or whatever you called it. What you see should give you warm fuzzies, something akin to what I'm reporting seeing in IntelliJ IDEA.

7. Prepare data flows to your processor.
8. Start your processor; the IDE debugger should pop up stopped at the first breakpoint.
9. Debug away.

I stumbled upon something peculiar in NiFi. I had been attaching an attribute to a flowfile in the session.read() call-back. The NiFi unit-testing framework tolerated this, but when I ran my processor for real, it blew chunks (IllegalStateException). I solved the problem by saving the new attribute and attaching it outside the call-back just before calling session.transfer().

The only reason I'm pointing this out is because I somewhat rely on unit testing and hope to catch this level of error/gotcha earlier than button-pushing testing.

Addendum posted 27 July 2017:

How to do the above using a tunnel...

Let's posit that you must debug your NiFi processor running on a remote server that's secure enough that you can't punch your debugging port through its firewall.

In this case, configure just as described above, but you'll be talking to your local host on the port you decide. Before editing your debug configuration, erect your IP tunnel as below. Let's pretend that the IP address of the remote host running NiFi is 10.10.9.158 and that my username there is russ:

# ssh -f -L 8000:10.10.9.158:8000 [email protected] -N

What this means is "set up my ssh (port 22) connection such that it handles all port 8000 traffic. At the other end, IP address 10.10.9.158, put that traffic back out to 8000. The 'tunnel' here is port 22 that carries the traffic meant for port 8000 carefully to and from the remote server without having to open port 8000 on that remote server (and punch a hole through its firewall, etc.) NiFi, configured in conf/bootstrap.conf to use port 8000 for debugging traffic, will answer back and the tunnel will handle the communication."

(This is an update of ...remote debugging of NiFi processors using IntelliJ IDEA written on Wednesday, 23 March 2016.)

I'm again working on remote debugging of NiFi processors using a much later version of IntelliJ IDEA. The instructions have not changed substantially, but this note updates them for accurate wording.

Steps to debugging a custom NiFi processor you've written. See this article to learn how remote debugging works in IntelliJ. The example uses IntelliJ to debug an application running in NiFi.

1. Edit conf/bootstrap.conf and simply uncomment the line that starts java.arg.debug. If you like, change the port number (address) to what you want to use in steps 2e or 2f below. Most often, I find myself using 8000.
2. Create a Debug configuration in IntelliJ IDEA, ...
   1. Do Run → Edit Configurations....
   2. Click the + icon.
   3. Choose Remote JVM Debug (because it's a remote debugging session you want to create).
   4. Give a Name:, something like "NiFi 1.26.1".
   5. Under Settings, change the Port: to 8000 (or to whatever you established in conf/bootstrap.conf). (Note: this number has nothing to do with the port number on which you've launching your browser, say http://localhost:8080/nifi.)
   6. If you're only debugging a single processor in a project with multiple modules, set the drop-down Search sources using module's classpath: to the module in which it lives.
3. Ensure that the NAR NiFi is using matches the code in which you plan to set breakpoints. (If it doesn't match, strange unworking symptoms may make you think that remote debugging does not work.)
4. Launch NiFi (or bounce it).
5. Set one or more breakpoints in your processor code. If you're not certain, get your bearings by setting a breakpoint you know you'll hit.
6. Launch the debugger: do Run → Debug and choose "Local NiFi" (or whatever you called it) from the list presented. This brings up the debugger and displays:

   "Connected to the target VM, address 'localhost:8000', transport: 'socket'
   

7. Prepare data flows for debugging your custom NiFi processor.
8. Start your processor; the IDE debugger should pop up stopped at the first breakpoint.
9. Merry debugging!