Don't remove the subdirectory resulting from the deployment. Just remove the WAR file and let Tomcat undeploy it. Doing this wrong leads confusingly to errors in catalina.out.

    [Tomcat] validateJarFile(servlet-api.jar) - jar not loaded. Offending class: javax/servlet/Servlet.class

    org.apache.catalina.loader.WebappClassLoader validateJarFile
    INFO: validateJarFile(\WEB-INF\lib\servlet-api.jar) - jar not loaded. See Servlet Spec 2.3, \
        section 9.7.2. Offending class: javax/servlet/Servlet.class.

Note that servlet-api.jar already exists in Tomcat. This message isn't anything more than a warning, but it's annoying. It means simply that you've included another instance of servlet-api.jar in your build path (see Eclipse Build Path or the .classpath file). Simply remove this JAR from your build letting Tomcat supply it.

You see this starting Tomcat. It's no more than a warning. In order to overcome it, you'd have to go to some trouble. Google for it.

    INFO: The APR based Apache Tomcat Native library which allows optimal performance in production \
        environments was not found on the java.library.path: \
        /home/russ/dev/jdk1.6.0_31/jre/lib/amd64/server:/home/russ/dev/jdk1.6.0_31/jre/lib/amd64:\
        /home/russ/dev/jdk1.6.0_31/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

See http://stackoverflow.com/questions/8716259/what-does-the-apr-based-apache-tomcat-native-library-was-not-found-mean.

What this means is:

1. You shouldn't worry about this during development; it's just a warning.
2. If this is a concern in production, and perhaps it should be, the only thing that can be done about it is to download Apache code and build a libnative.so for the platform that's going to execute your production application. See http://tomcat.apache.org/tomcat-6.0-doc/apr.html#Installation.

Tomcat's server.xml file is amended thus in the face of implementing HTTPS. The part in bold type is what's added to an otherwise stock server.xml:

    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               redirectPort="8443" />

    <!-- We added this next paragraph in recognition that Big IP will send
         secure (formerly encrypted) requests through port 8081 and calling
         ContainerRequest.isSecure() will return true.
    -->
    <Connector port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               URIEncoding="UTF-8"
               scheme="https"
               secure="true"
    />

If Tomcat's handling this by itself, in the situation where there is no Big IP or load-balancer, the traffic comes in encrypted over 443 (instead of 8081) and Tomcat decrypts it.

There is a second part to this equation. The following section(s), if present, must be removed (commented out) from server.xml:

     <!-- Define a SSL HTTP/1.1 Connector on port 8443
          This connector uses the JSSE configuration, when using APR, the
          connector should be using the OpenSSL style configuration
          described in the APR documentation -->
     <!-- Comment this out as we don't use SSL; instead we get secure traffic over 8081
     <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
                maxThreads="150" scheme="https" secure="true"
                clientAuth="false" sslProtocol="TLS" />
     -->

The lines being commented out come by default in the stock Tomcat server.xml. The traffic is unencrypted by the load balancer and passed over port 8081 instead of 8443 and we want to stop listening on port 8443 which will otherwise give us keystore errors like:

    Aug 09, 2013 4:55:12 PM org.apache.tomcat.util.net.jsse.JSSESocketFactory getStore
    SEVERE: Failed to load keystore type JKS with path /usr/share/tomcat6/.keystore due to /usr/share/tomcat6/.keystore (No such file or directory)
    java.io.FileNotFoundException: /usr/share/tomcat6/.keystore (No such file or directory)
            at java.io.FileInputStream.open(Native Method)
            at java.io.FileInputStream.(FileInputStream.java:138)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:405)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:296)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:544)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:481)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:156)
            at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
            at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:176)
            at org.apache.catalina.connector.Connector.initialize(Connector.java:1049)
            at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
            at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

    Aug 09, 2013 4:55:12 PM org.apache.coyote.http11.Http11Protocol init
    SEVERE: Error initializing endpoint
    java.io.FileNotFoundException: /usr/share/tomcat6/.keystore (No such file or directory)
            at java.io.FileInputStream.open(Native Method)
            at java.io.FileInputStream.(FileInputStream.java:138)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:405)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:296)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:544)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:481)
            at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:156)
            at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
            at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:176)
            at org.apache.catalina.connector.Connector.initialize(Connector.java:1049)
            at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
            at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

    Aug 09, 2013 4:55:12 PM org.apache.catalina.core.StandardService initialize
    SEVERE: Failed to initialize connector [Connector[HTTP/1.1-8443]]
    LifecycleException:  Protocol handler initialization failed: java.io.FileNotFoundException: /usr/share/tomcat6/.keystore (No such file or directory)
            at org.apache.catalina.connector.Connector.initialize(Connector.java:1051)
            at org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
            at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:838)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
            at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
            at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
            ...

This assumes that on an Ubuntu Precise Server installation, only the OpenJDK is installed, which it is by default. This example assumes Java 1.6, call it jdk1.6.0_38 for the sake of our illustration.

There are reasons you want to do this. One is the ability to run more competent encryption/hashing.

First, you can see which JVM is being used by Tomcat via this command:

    $ ps -ef | grep [t]omcat

Likely, it will be /usr/lib/jvm/default-java/bin/java. If you list this path, you'll see that it's the non-Sun OpenJDK.

    $ ll /usr/lib/jvm/default-java
    lrwxrwxrwx 1 root root 24 Jan 29 06:30 /usr/lib/jvm/default-java -> java-1.6.0-openjdk-amd64/

1. Download the Sun JRE or JDK from http://java.sun.com; download the .bin file. At time of writing, once you've reached this page, you'd hover over DOWNLOADS and choose Java for Developers under "Popular Downloads".
2. Copy the .bin over to your server.
3. Copy the .bin file to /usr/local, make it executable and explode it.
4. Go to /usr/local.
5. Create a new link, jdk1.6.0_38 to point at the new JDK/JRE or just move the whole JDK/JRE to this path.
6. Own the exploded download (chown -R root:root jdk1.6.0_38).
7. Go to /usr/lib/jvm.
8. Create a link named java-6-oracle to /usr/local/jdk1.6.0_38.
9. Create a new link named default-java (or, replace the one that's there) to point at ./java-6-oracle. This is the key to getting Tomcat to use it (examine /etc/init.d/tomcat6, search for "default-java" and you'll understand).

At this point, bounce Tomcat and you'll see that it's running atop the real, Sun JDK/JRE.

    $ /etc/init.d/tomcat6 restart
    $ ps -ef | grep [t]omcat
    (stuff similar to what you saw before except...)
    $ ll /usr/lib/jvm/default-java
    lrwxrwxrwx 1 root root 24 Jan 29 06:30 /usr/lib/jvm/default-java -> java-6-oracle/

If one leg of your load-balanced application is not up, you could experience peculiarities like missing CSS, images, etc.

To see whether Tomcat is running on 8080, do this:

    root@app-1:# lsof -i :8080
    COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
    java    916 tomcat6   33u  IPv6   7764      0t0  TCP *:http-alt (LISTEN)

If you're out of space or even catalina.out gets much over 500Mb, you could fail to deploy new or redeploy existing applications because of it.

Good rules of thumb for this configuration are:

• Roll log files every 500 Mb.
• Keep only 5 files back.

There's lots of idiocy out there when you look for this. Most of it comes from a Windoz orientation. However, others seem to answer not knowing that the user tomcat6 isn't something you can "become" (via su), have a "home" directory, etc.

At least for Ubuntu and Mint, the place to look is /etc/default/tomcat6, a file that contains a lot of settings or commented-out potential settings.

If you want to switch what version of Java your Tomcat installation is running on, there are two, good ways. Let's imagine you want to begin running Java 7 instead of 6.

1. As just alluded, tweak the value for JAVA_HOME in /etc/default/tomcat6 to point at the Java 7 you installed.
 
2. Determine what version of Java is available and where it's running. You should see something like this:

       # which java
       /usr/bin/java
       # java -version
       java version "1.7.0_21"
       OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-0ubuntu0.12.04.1)
       OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
   

    
3. Install, e.g.: openjdk7, then uninstall openjdk6. Don't touch any definitions of JAVA_HOME anywhere. This is the approach to take if you're dealing with a VM running nothing but Tomcat (and your web application suite).

       # java -version
       java version "1.6.0_27"
       OpenJDK Runtime Environment (IcedTea6 1.12.5) (6b27-1.12.5-0ubuntu0.12.04.1)
       OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
       # apt-get update
       # apt-get install openjdk-7-jre
       (installs latest, known Java 7)
       # apt-get purge openjdk-6-jre-headless
       (removes Java 6 and dependencies; lose system notion of Java)
   

   Ordinarily, however, on Ubuntu server systems I've installed using mostly defaults I have not had anything to do past this point.

4. You should examine /etc/alternatives to see if everything is good.

       # cd /etc/alternatives
       /etc/alternatives # ln -s /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/javaws*
       (updates system notion of Java which was lost)
       # java -version
       java version "1.7.0_21"
       OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-0ubuntu0.12.04.1)
       OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)
   

   If everything is okay, you should see only Java 7 here:

       root@taliesin:/etc/alternatives# ll java*
       lrwxrwxrwx 1 root root 46 Nov 12 21:01 java -> /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java*
       lrwxrwxrwx 1 root root 56 Nov 12 21:01 java.1.gz -> /usr/lib/jvm/java-7-openjdk-amd64/jre/man/man1/java.1.gz
   

   If this wasn't enough, note that replacing Java 6 with 7 is not for the faint-hearted. You need to do more than simply install openjdk/jre 7 or Sun Java 7. You also have to update the Java "alternatives" to make it look more or less like the above. (Or see below for a more twisted result.)

^* Despite complaints and claims to the contrary out there, there is no way update-java-alternatives will update the javaws links; you have to do it by hand. However, you need not do it if this link was not there in the first place.

More notes

On occasion, the above is insufficient and you can still see this in the Tomcat Web Application Manager, the wrong JVM:

(To get into the manager go to http://hostname:port/manager/html. If you don't know how to respond to the authentication dialog, see below.)

I thought I had laid to rest the question of which JVM would hold sway on the server host. Following is a picture I hope will be a thousand words. Please note especially the lines in bold which describe the state of things after I fixed the remaining trouble.

root@acme:/usr/lib/jvm# ll
total 24
drwxr-xr-x  5 root root 4096 Nov  6 16:06 ./
drwxr-xr-x 64 root root 4096 Nov  6 14:18 ../
lrwxrwxrwx  1 root root   21 Nov  6 16:06 default-java -> java-7-openjdk-amd64//
lrwxrwxrwx  1 root root   20 Jul 15 19:46 java-1.7.0-openjdk-amd64 -> java-7-openjdk-amd64/
-rw-r--r--  1 root root 2439 Jul 15 19:46 .java-1.7.0-openjdk-amd64.jinfo
lrwxrwxrwx  1 root root   15 Jan 25  2013 java-6-sun -> /usr/local/java/
drwxr-xr-x  5 root root 4096 Nov  6 14:17 java-7-openjdk-amd64/
drwxr-xr-x  3 root root 4096 Nov  6 14:17 java-7-openjdk-common/
drwxr-xr-x  2 root root 4096 Feb 26  2012 java-7-oracle/
lrwxrwxrwx  1 root root   10 Jan 25  2013 old-default-java -> java-6-sun/

Links to peruse:

• http://askubuntu.com/questions/21131/how-to-correctly-remove-openjdk-openjre-and-set-sunjdk-sunjre-as-default
• http://unix.stackexchange.com/questions/53594/java-version-for-tomcat

There are many reasons this can happen. One that might escape is that the application was deployed (i.e.: it shows up in /var/lib/tomcat6/webapps), but it failed to start up. Examine the log, /var/log/tomcat6/catalina.out to see why. Perhaps you were trying to connect to your database node(s), but had forgotten to update the application server's /etc/hosts file with the mappings to those hosts and they aren't know to DNS. Etc.

In catalina.out, you might find mysteriously, one of...

    SEVERE: Error filterStart
    SEVERE: Error listenerStart

...when you try to start your application. By default, Tomcat won't tell you intelligently what's happening. It won't even tell you which filter or listener is failing. The solution is very simple:

1. Under Tomcat's webapps folder, under your deployed application, create WEB_INF/class/logging.properties—even if you've already got log4j.properties, etc. there.
2. Add the following content to this file:

       org.apache.catalina.core.ContainerBase.[Catalina].level = ALL
       org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler
   

3. Clear out catalina.out if you like to squint through less output.
4. Bounce Tomcat:

       $ service tomcat6 restart
   

You should see a proper stacktrace with all the verbosity you need to sort this problem out.

...after installing Tomcat 6. Old Apache log4j like Apache Tomcat 5.5 Servlet/JSP Container: Logging in Tomcat says to use statements like

    log4j.appender.R.File=${catalina.home}/logs/myapp.log

...where myapp.log is the name you invent for the logfile different from catalina.out.

This won't work, at very least, on Debian platforms where ${catalina.home} points at /usr/share/tomcat6, which has no logs subdirectory or link to /var/log/tomcat6 where catalina.out lives. Use ${catalina.base} instead; it points at /var/lib/tomcat6 which has logs as a symbolic link to /var/log/tomcat6.

When you see something like...

    log4j:ERROR setFile(null,true) call failed.
    java.io.FileNotFoundException: /usr/share/tomcat6/logs/myapp.log (No such file or directory)
            at java.io.FileOutputStream.open(Native Method)
            at java.io.FileOutputStream.(FileOutputStream.java:212)
            at java.io.FileOutputStream.(FileOutputStream.java:136)
            at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
            at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
            at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
            at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
            at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
            at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
            at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809)
            at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735)
            at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615)
            at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502)
            at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547)
            at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483)
            at org.apache.log4j.LogManager.(LogManager.java:127)
            at org.apache.log4j.Logger.getLogger(Logger.java:117)
            at com.hp.web.util.ApplicationProperties.(ApplicationProperties.java:17)
            at com.hp.web.controller.AppInitializer.contextInitialized(AppInitializer.java:20)
            at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
            at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
            at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
            at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
            at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
            at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
            at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
            at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
            at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385)
            at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:306)
            at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
            at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1389)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1653)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1662)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1642)
            at java.lang.Thread.run(Thread.java:722)

...it's because Tomcat is unable to create /usr/share/tomcat/logs/myapp.log. This is because the path to it doesn't exist.

To get into the manager go to http://hostname:port/manager/html. If you don't know what to respond to the authentication dialog, it's because /etc/tomcat6/tomcat-users.xml has nothing in it. You'll want to add...

        .
        .
        .
        <role rolename="manager"/>
        <role rolename="manager-gui"/>
        <role rolename="admin"/>
        <user username="user" password="password" roles="admin,manager,manager-gui"/>
    </tomcat-users>

...inside the <tomcat-users> tag. Replace user and password with something a little more secure.

Add a file called ROOT.xml in $CATALINA_HOME/conf/Catalina/localhost/

ROOT.xml will override the default settings for the root context of the Tomcat installation for that engine and host (Catalina and localhost).

Enter the following to the ROOT.xml file:

<Context docBase="yourApp" path="" reloadable="true" />

Here, yourApp is the name of your application.

And there you go, your application is now the default application and will show up on http://localhost:8080.

However, there is one side effect; your application will be loaded twice. Once for localhost:8080 and once for localhost:8080/yourApp. To fix this you can put your application OUTSIDE $CATALINA_HOME/webapps and use a relative or absolute path in the ROOT.xml's docBase tag. Something like this:

<Context docBase="/opt/mywebapps/yourApp" path="" reloadable="true" />

Individual Context elements may be explicitly defined:

• In an individual file at /META-INF/context.xml inside the application files.

In Tomcat's server.xml file, at the bottom, you can awaken ('cause it ships commented out) the access log by uncommenting and configuring it. Here's something I did one day, with consideration for Tomcat 7 and solving a problem explained in the additional comment:

  <!-- Access log processes all example.
       Documentation at: /docs/config/valve.html -->
  <!--
       Note: this is uncommented giving us 1 second granularity in the
       log entries, see
       http://tomcat.apache.org/tomcat-6.0-doc/config/valve.html

       Millisecond granularity will (only) come after moving to Tomcat 7.
       The "common" pattern, even in Tomcat 7, does not include this, so
       it will explicitly become "%h %l %u %{msec}t "%r" %s %b". See
       https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html
    -->
  <Valve className="org.apache.catalina.valves.AccessLogValve"
         directory="/home/fs/log"
         prefix="tomcat-access."
         suffix="log"
         pattern="%h %{x-forwarded-for}i; %H %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" %Dms"
         resolveHosts="false"
         rotatable="false"
         checkExists="true"
              />

"SEVERE: ERROR LISTENERSTART" AND "SEVERE: ERROR FILTERSTART" TOMCAT ERROR MESSAGES

The errors being sparse, I add logging.properties to src with contents:

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

From a certain Faheem Sohail...

In HTTP 1.1, all connections between the browser and the server are considered persistent unless declared otherwise. Persistence, in this context, means to use a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair.

In Tomcat, the default HTTP connector is blocking and follows a one thread per connection model. This means that in order to serve 100 concurrent users, it requires 100 active threads. We end up wasting resources (the thread) because connections may not be used heavily, but just enough to avoid a timeout.

Opposed to this is the relatively new NIO or non blocking connector. This connector has a couple of poller threads used to keep the connection alive for all connected users while worker threads are called whenever data (a new HTTP request) is available. This model leads to a much better sharing of resources (threads) and a larger number of concurrent users can be served from the same server.

In order to configure tomcat to use the non-blocking NIO connector instead of the default blocking BIO one simply change the value of the protocol attribute of the connector tag in the server.xml from HTTP/1.1 to org.apache.coyote.http11.Http11NioProtocol.

To verify that you indeed are using the NIO connector, take a look at the start-up logs. You should see lines similar to this.

Mar 28, 2014 3:59:04 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8080"]
Mar 28, 2014 3:59:04 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector

Use VisualVM to look at the threads being created in both cases. You’ll find NIO to use threads much more efficiently.

This is also a good example of how to find the JAR containing a symbol that will satisfy what's missing in your Java code. Note that the convoluted nature of this search, which has the added complication that this symbol is only provided and not linked in (it's Tomcat that will load my servlet and provide the JAR containing the symbol), demonstrates that you must be careful where you get the symbol. www.findjar.com is a very useful site, but you must be certain which JAR is the right source from among the many it often suggests, i.e.: there's a sort of Zen to it that comes with experience in the sort of Java programming that you're doing.

To code to get the incoming request for a POST operation in my restlet, ...

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.HttpHeaders;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import javax.servlet.http.HttpServletRequest;

public class Restlet
{
  @POST
  @Consumes( MediaType.APPLICATION_FORM_URLENCODED )
  @Produces( MediaType.TEXT_XML )
  public Response postPatientData( @Context HttpServletRequest request, @Context HttpHeaders header )
  {
    ...
  }

...I needed to consume this symbol from Tomcat. Here's how I determined that:

1. Google "what jar HttpServletRequest".
2. http://www.findjar.com/class/javax/servlet/http/HttpServletRequest.html suggested a number of JARs.
3. I looked in my Tomcat download, ~/dev/apache-tomcat-9.0.7/lib and found servlet-api.jar that matches one of the JARs the find-jar site suggested. This happesn to be the server container I'm using in my restlet, so, I conclude that I'll get HttpServletRequest from there.
4. I use the (Linux) Nemo filesystem browser to navigate to and open (double-click) ~/dev/apache-tomcat-9.0.7/lib/servlet-api.jar. This launches Archive Manager which shows me inside this JAR without the need to explode it manually. Where it not for this, I'd have to resort to doing what I describe in Viewing the contents of a JAR.
5. I verify that the symbol, HttpServletRequest, can be found on the package path, javax.servlet.http, by clicking down through these three subdirectories to see HttpServletRequest.class.
6. I look inside META-INF/MANIFEST.MF for the version of this JAR, 4.0:

   Name: javax/servlet/
   Specification-Title: Java API for Servlets
   Specification-Version: 4.0
   Specification-Vendor: Sun Microsystems, Inc.
   Implementation-Title: javax.servlet
   Implementation-Version: 4.0.FR
   Implementation-Vendor: Apache Software Foundation
   

7. I Google "maven servlet-api" and see "Maven Repository: javax.servlet >> servlet-api"; I click below on 4.0.0. This gives me the Maven-dependency paragraph:

   <dependency>
     <groupId>javax.servlet</groupId>
     <artifactId>javax.servlet-api</artifactId>
     <version>4.0.0</version>
     <scope>provided</scope>
   </dependency>
   

8. I paste this into my project's pom.xml and refresh Maven (a function of IntelliJ IDEA) whereupon the red for this symbol disappears.

Let's look at a rather stand-alone way (not as a service) to set up and run Tomcat under Linux. I'm doing this on Linux Mint (Ubuntu).

1. For much of this, you'll need to have root access:

   $ sudo bash
   # whoami
   root
   

2. Add user tomcat to system; I chose tomcat as the password too since this is only a tutorial:

   # adduser tomcat
   Adding user `tomcat' ...
   Adding new group `tomcat' (1003) ...
   Adding new user `tomcat' (1003) with group `tomcat' ...
   Creating home directory `/home/tomcat' ...
   Copying files from `/etc/skel' ...
   Enter new UNIX password: tomcat
   Retype new UNIX password: tomcat
   passwd: password updated successfully
   Changing the user information for tomcat
   Enter the new value, or press ENTER for the default
   	Full Name []: Tomcat
   	Room Number []:
   	Work Phone []:
   	Home Phone []:
   	Other []:
   Is the information correct? [Y/n] Y
   nargothrond ~ # su - tomcat
   tomcat@nargothrond ~ $ pwd
   /home/tomcat
   

3. Download Tomcat 9 from here. Under Core, click on the tarball.

4. Explode the tarball:

   tomcat@nargothrond ~ $ tar -zxf apache-tomcat-9.0.8.tar.gz
   tomcat@nargothrond ~ $ ll -d apache*
   drwxrwxr-x 9 tomcat tomcat    4096 Jun 12 14:18 apache-tomcat-9.0.8/
   -rw-rw-r-- 1 tomcat tomcat 9818695 Jun 12 14:08 apache-tomcat-9.0.8.tar.gz
   

5. Modify ~/.bash_profile to define the Catalina root:

   tomcat@nargothrond ~ $ vi ~/.bash_profile
   
   export CATALINA_HOME=/home/tomcat/apache-tomcat-9.0.8
   

6. Verify that all this is set up:

   tomcat@nargothrond ~ $ exit
   nargothrond ~ # su - tomcat
   tomcat@nargothrond ~ $ set | grep CATALINA
   CATALINA_HOME=/home/tomcat/apache-tomcat-9.0.8
   

7. Start Tomcat and verify that it's running:

   tomcat@nargothrond ~ $ $CATALINA_HOME/bin/catalina.sh start
   Using CATALINA_BASE:   /home/tomcat/apache-tomcat-9.0.8
   Using CATALINA_HOME:   /home/tomcat/apache-tomcat-9.0.8
   Using CATALINA_TMPDIR: /home/tomcat/apache-tomcat-9.0.8/temp
   Using JRE_HOME:        /usr
   Using CLASSPATH:       /home/tomcat/apache-tomcat-9.0.8/bin/bootstrap.jar:/home/tomcat/apache-tomcat-9.0.8/bin/tomcat-juli.jar
   Tomcat started.
   tomcat@nargothrond ~ $ ps -ef | grep -i [t]omcat
   root     26366 24746  0 14:28 pts/3    00:00:00 su - tomcat
   tomcat   26374 26366  0 14:28 pts/3    00:00:00 -su
   tomcat   26459     1  2 14:31 pts/3    00:00:03 /usr/bin/java -Djava.util.logging.config.file=/home/tomcat/apache-tomcat-9.0.8/co...
   tomcat   26607 26374  0 14:33 pts/3    00:00:00 ps -ef
   tomcat   26608 26374  0 14:33 pts/3    00:00:00 grep -i [t]omcat
   

8. Examine Tomcat log files:

   tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ head --lines=13 catalina.out
   12-Jun-2018 14:31:25.718 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:         Apache Tomcat/9.0.8
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:           Apr 27 2018 19:32:00 UTC
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:          9.0.8.0
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:                Linux
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:             4.13.0-36-generic
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:           amd64
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:              /usr/lib/jvm/java-8-openjdk-amd64/jre
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:            1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:             Oracle Corporation
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:          /home/tomcat/apache-tomcat-9.0.8
   12-Jun-2018 14:31:25.719 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:          /home/tomcat/apache-tomcat-9.0.8
   12-Jun-2018 14:31:25.720 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/tomcat/apach...
   12-Jun-2018 14:31:25.720 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassL...
   
   tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ tail --lines=3 catalina.out
   12-Jun-2018 14:31:26.174 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
   12-Jun-2018 14:31:26.179 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
   12-Jun-2018 14:31:26.181 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 389 ms
   

9. Go back up to the root:

   tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ cd ..
   tomcat@nargothrond ~/apache-tomcat-9.0.8/logs $ ../bin/catalina.sh stop
   

10. You only need one of these set in ~/.bash_profile if you want to avoid using whatever installed version of Java you have on your host:
    • JAVA_HOME
    • JRE_HOME
    Map it to the root of your alternative JDK, etc. installation.

11. Last, check out Apache Tomcat's default home page: http://localhost:8080

    

Let's look at a way to install Tomcat 9 as a service on CentOS. I nevertheless did all of this (minus the firewalld stuff) successfully on Linux Mint (Ubuntu). This is the result of some research: as of this writing, there was no packaging by which you can get Tomcat 9 from yum, apt, etc. So, this is how they do it. (I read probably six different posts about this.)

1. For much of this, you'll need to have root access:

   $ sudo bash
   # whoami
   root
   

2. Create a system user and group before installing Tomcat 9. This disables shell access, creates a new user tomcat and new group tomcat, then puts the user into the group. /opt/tomcat becomes the home directory:

   # groupadd tomcat
   # useradd -s /bin/false -g tomcat -d /opt/tomcat tomcat
   

3. Download and install Tomcat 9. You can't use yum (or apt-get).

   # cd /opt
   # wget http://www.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
   --2018-06-12 15:18:34--  http://www.apache.org/dist/tomcat/tomcat-9/v9.0.8/bin/apache-tomcat-9.0.8.tar.gz
   Resolving www.apache.org (www.apache.org)... 40.79.78.1, 95.216.24.32, 2a01:4f9:2a:185f::2
   Connecting to www.apache.org (www.apache.org)|40.79.78.1|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 9818695 (9.4M) [application/x-gzip]
   Saving to: 'apache-tomcat-9.0.8.tar.gz'
   
   apache-tomcat-9.0.8.tar.gz   100%[========================================>]   9.36M  4.94MB/s    in 1.9s
   2018-06-12 15:18:36 (4.94 MB/s) - 'apache-tomcat-9.0.8.tar.gz' saved [9818695/9818695]
   # tar -zxf apache-tomcat-9.0.8.tar.gz       # explode tarball to subdirectory
   # rm apache-tomcat-9.0.8.tar.gz             # remove unneeded tarball
   # mv apache-tomcat-9.0.8 tomcat             # rename tomcat subdirectory
   # chown -R tomcat:tomcat tomcat/*           # fix up ownership
   

4. Set up the systemd service script. This is something you must do and a configuration opportunity. The biggest example is memory—as configured in the CATALINA_OPTS line. See this discussion.

   # vim /etc/systemd/system/tomcat.service
   [Unit]
   Description=Apache Tomcat 9 Servlet Container
   After=syslog.target network.target
   
   [Service]
   Type=forking
   
   User=tomcat
   Group=tomcat
   
   Environment=CATALINA_PID=/opt/tomcat/tomcat.pid
   Environment=CATALINA_HOME=/opt/tomcat
   Environment=CATALINA_BASE=/opt/tomcat
   Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
   Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'
   
   ExecStart=/opt/tomcat/bin/startup.sh
   ExecStop=/opt/tomcat/bin/shutdown.sh
   Restart=on-failure
   
   User=tomcat
   Group=tomcat
   UMask=0007
   RestartSec=10
   Restart=always
   
   [Install]
   WantedBy=multi-user.target
   

5. Run the following commands to cause Tomcat to start as a service upon boot:

   # systemctl daemon-reload                   # to reload the systemd dæmon
   # systemctl start tomcat                    # to start Tomcat
   # systemctl enable tomcat                   # to enable Tomcat
   # netstat -plntu                            # to check to see what port Tomcat's running on (default is 8080)
   # systemctl status tomcat                   # to check Tomcat's status
   

6. Add the following lines to /opt/tomcat/conf/tomcat-users.xml just before the </tomcat-users> closing element:

   <role rolename="manager-gui" />
   <user username="admin"* password="password"* roles="manager-gui,admin-gui" />
   

   ^* I used tomcat and tomcat.

7. Comment out the following line(s) from /opt/tomcat/webapps/manager/META-INF/context.xml (see Remote access below). This is only so that you can reach the Tomcat Web Application Manager remotely. It has nothing to do with whether you can reach the application you're hosting using Tomcat. (For that, see Hosted application access below.)

   18 <Context antiResourceLocking="false" privileged="true">
   19   <!-- <Valve className="org.apache.catalina.valves.RemoteAddrValve"
   20         allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" /> -->
   21   <Manager sessionAttributeValueClassNameFilter="java\.lang\.(?:Boolean|Integer|Long|Num...
   22 </Context>
   

8. /opt/tomcat/webapps/host-manager/META-INF/context.xml governs local and remote access to the Tomcat Virtual Host Manager application. See a note in Remote access below to understand more about this.

9. There might be some value to adding the following line to /opt/tomcat/conf/context.xml. This might speed Tomcat up a bit by changing what it does when it loads applications dropped into the webapps subdirectory:

     <JarScanner scanClassPath="false" />
   </Context>
   

10. ---

    Important note: I tried firewalld once; it totally destroyed my host. It's very intrusive and I couldn't unwind it from everything it invaded. I suggest something else (and since I'm not a Red Hat sort of guy, you're on your own there).
    
    

    Configure the firewall (example using firewalld and default port 8080):

    # yum install firewalld                                       # install firewalld
    # systemctl start firewalld                                   # start firewalld
    # systemctl enable firewalld                                  # enable firewalld
    # firewall-cmd --zone=public --permanent --add-port=8080/tcp  # configure firewalld
    # firewall-cmd --reload                                       # (ibid)
    # firewall-cmd --list-ports                                   # verify configuration of firewalld
    # firewall-cmd --list-services                                # ibid
    

    ---

11. Check out web access from localhost and from a remote host:
    http://localhost:8080
    http://10.10.10.6:8080
    The latter did not work for me, but as it's not pressing (yes, it should be); I'm letting this slide for now as I'm busy. I'll come back later and say why and what I did to fix it. Maybe it's the stuff I commented out in context.xml.

12. Troubleshooting: if the service doesn't start, try

    # systemctl status tomcat.service
    

    and if you see something like:

    Jun 12 16:09:46 nargothrond systemd[1]: Starting Apache Tomcat 9 Servlet Container...
    Jun 12 16:09:46 nargothrond startup.sh[32038]: touch: cannot touch '/opt/tomcat/logs/catalina.out': Permission denied
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Control process exited, code=exited status=1
    Jun 12 16:09:46 nargothrond systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Unit entered failed state.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Failed with result 'exit-code'.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Service hold-off time over, scheduling restart.
    Jun 12 16:09:46 nargothrond systemd[1]: Stopped Apache Tomcat 9 Servlet Container.
    Jun 12 16:09:46 nargothrond systemd[1]: tomcat.service: Start request repeated too quickly.
    Jun 12 16:09:46 nargothrond systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    

    Ensure that /opt/tomcat/logs isn't full of log files that are owned by root:root (because of early, imprecise attempts at running Tomcat). If, instead, you see this:

    Jun 14 18:35:56 psa98.acme.io systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Unit tomcat.service entered failed state.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: tomcat.service failed.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: tomcat.service holdoff time over, scheduling restart.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: start request repeated too quickly for tomcat.service
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Failed to start Apache Tomcat 9 Servlet Container.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: Unit tomcat.service entered failed state.
    Jun 14 18:35:56 psa98.acme.io systemd[1]: tomcat.service failed.
    

    I found that this cleared itself by doing systemctl restart tomcat.service. However, another time, it appeared because the download was itself broken which was discovered as soon as an attempt was made to look at logs/catalina.out, which didn't exist. The correctly exploded download must be contained on the path /opt/tomcat with various subdirectories like bin, conf, logs, etc.

Think your servlet is just perfect, it works in IntelliJ IDEA, all your ducks are in line, but you get HTTP status or error codes of 404, 406, 415 or 400?

In Tomcat, the application context root (that is, what comes immediately after the port number in the URL) is the name of the WAR file. This isn't true for all that's in ROOT, which is just /.

So, this is the answer. It was suggested by something my friend Scott said to me:

What is the URL used to try to access the endpoint? From the configuration you posted, it looks like it would be: http://localhost:8080/mdht-restlet/mdht-restlet

I'm going to attempt to quantify it. Why did this not mess up in IntelliJ IDEA's deployment, I don't know yet. Now some of this is arbitrary and there is flexibility, but following this will work.

1. web.xml. Start here. Make servlet-name identical to the servlet's project name (in the highlighted lines below; this code is also not complete):

     <display-name>mdht-restlet</display-name>
   
     <servlet>
       <servlet-name>mdht-restlet</servlet-name>
   
       <init-param>
         <param-name>com.sun.jersey.config.property.packages</param-name>
         <param-value>com.acme.servlet</param-value>
       </init-param>
   
       <init-param>
         <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
         <param-value>true</param-value>
       </init-param>
     </servlet>
   
     <servlet-mapping>
       <servlet-name>mdht-restlet</servlet-name>
       <url-pattern>/*</url-pattern>
     </servlet-mapping>
   
   

2. You can't name the Java source of the servlet this, but I keep it the same plus conventions: MdhtRestlet.java.

3. Inside the servlet code, using the @Path annotation, indicate the path from the root to the actions named for each method. I don't have any deeper naming, so I just have no additional paths. But, when I had this problem (after 5+ years of not doing web applications), I was using @Path( "/mdht-restlet" )—hence Scott's comment to me (as reported above). Do this instead:

   @Path( "" )
   public class MdhtRestlet
   {
     @POST
     public Response postPatientData( ... ) { }
   
     @GET
     public String getStatusInPlainText() { }
   }
   
   

   This makes the URL to reach the servlet methods, whether POST or GET, always (and just) http://localhost:8080/mdht-restlet.

4. Of course, this could be very much more complex—and would be in a complex application like Snapfish's accountmgr back when I created that:

   @Path( "/account" )
   public class AccountMgr
   {
     @POST
     @PATH( "/user" )
     public Response createAccount( ... ) { }
   
     @GET
     @PATH( "/user" )
     public String getAccount() { }
   
     @PUT
     @PATH( "/user/update" )
     public String modifyAccount() { }
   
     @DELETE
     @PATH( "/user/delete" )
     public String deleteAccount() { }
   }
   
   

   Thus, the URLs will be (in above order):
   1. POST http://localhost:8080/accountmgr/account/user
   2. GET http://localhost:8080/accountmgr/account/user
   3. PUT http://localhost:8080/accountmgr/account/user/update
   4. DELETE http://localhost:8080/accountmgr/account/user/delete

https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000021519-Tomcat-deployment-and-use-in-IDEA-works-but-404-when-deployed-to-server

(${CATALINA_ROOT}/conf/context.xml is a file in your neighborhood (à la Mr. Rogers). It's not used for anything in this note.)

This setting, in ${CATALINA_ROOT}/webapps/manager/META-INF/context.xml,

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
   allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />

...will enforce access only for the clients connecting from localhost.

(Remember, we're talking about remote access here using the org.apache.catalina.valves.RemoveAddrValve class. Besides these "valve" details to consider, you must also punch a hole in your firewall for whatever port you're running Tomcat on as defined in conf/server.xml.)

This comes by default in the installation. So, to permit access from the outside, comment the <Valve... /> element out.

To permit unrestricted access for clients from localhost, but require authentication for all other (remote) clients, which must come in over port 8443, add this:

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       addConnectorPort="true"
       allow="127\.\d+\.\d+\.\d+;\d*|::1;\d*|0:0:0:0:0:0:0:1;\d*|.*;8443" />
<Valve className="org.apache.catalina.authenticator.BasicAuthenticator" />

The basic authenticator will work off the details you (may have) added to ${CATALINA_ROOT}/conf/tomcat-users.xml. Look elsewhere in these notes for this file.

To enable unrestricted access via port 8080:

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       addConnectorPort="true"
       allow=".*;8080" />

The configuration above really belongs to the Tomcat Web Application Manager:

Notes on accessing Tomcat "host" Manager

By default, the Host Manager, configured in ${CATALINA_ROOT}/webapps/host-manager/META-INF/context.xml, is only accessible from a browser running on the same host as Tomcat. If you can't live with this restriction, you'll need to edit this file to allow all IP addresses (or whatever ones you want). This file does not govern access to applications Tomcat is hosting, but only its manager interface.

To allow this access, change to:

<Context antiResourceLocking="false" privileged="true">
  <Valve className="org.apache.catalina.valves.RemoteAddrValve" allow=".*" />
</Context>

This configuration belongs to the Tomcat Virtual Host Manager:

Not sure what I was thinking in the note above, but to accesss anything from the Tomcat Manager (http://host:8080), of all the possible "solutions" I found, only doing this to

${CATALINA_HOME}/webapps/manager/META-INF/context.xml

...allowed me to reach through any of the buttons in the manager from the browser on my development host on the remote server running my target application on Tomcat. Basically, I added the comment characters in the paragraph below to what was already there:

<Context antiResourceLocking="false" privileged="true">
  <CookieProcessor className="org.apache.tomcat.util.http.Rfc6265CookieProcessor"
                   sameSiteCookies="strict" />
  <!--
  <Valve className="org.apache.catalina.valves.RemoteAddrValve"
         allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />
    -->
  <Manager sessionAttributeValueClassNameFilter="java\.lang\.(?:Boolean|Integer|Long|Number|String)|org\.apache\... (truncated)
</Context>

Caveats

Obviously, this opens that host's Tomcat Manager to any Tom, Dick or Harry that can reach it to gain access. In my case, I don't care because I'm behind a firewall and I put nothing sensitive on this (testing) server. I googled for and saw several "safer" and "more correct" ways of doing this, including adding a file, manager.xml to /opt/tomcat/conf/Catalina/localhost with a regular expression in the RemoteAddrValve's allow attribute, but I could not get that to work. Nor could I get such a modification to work in the solution above in being more clever than just commenting it out.

You've developed a web application that you deploy to Tomcat and it works as you access it locally, but when you try to access it remotely you cannot.

First, this is likely because your firewall, for the port on which Tomcat is configured in conf/server.xml is not open (to the world). For example, if I have Tomcat running on port 8080:

# netstat -pln | grep 8080                            # see that it's not open
# iptables -A INPUT -p tcp --dport 8080 --jump ACCEPT # make a hole
# netstat -pln | grep 8080                            # see if it's open

This did it for me, however...

Second, by default, Tomcat disables access from outside IP addresses. You may (or not—I didn't have to) add the following new line to conf/server.xml:

<Connector port="8080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           URIEncoding="UTF-8"
           redirectPort="8443"
           useIPVHosts="true" />

...and, last, sometimes even:

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
       prefix="localhost_access_log" suffix=".txt"
       pattern="%h %l %u %t "%r"e; %s %b"
       resolveHosts="true" />

Both...

• are open source
• written in Java
• implement the Servlet and JavaServer Pages specifications
• widely used in production environments

In embedded constructions, for example, Apache NiFi uses an embedded Jetty to support its UI and its ReST API, Jetty may have an edge.

Any (and all) application(s) running under Tomcat use Tomcat's Java virtual machine (JVM) including memory.

The stack- and heap memory established for Tomcat are defined in environment variable, CATALINA_OPTS, as:

• -Xms=512M   (stacksize starting amount)
• -Xmx=1024M (heap maximum)

The above are Tomcat 9's defaults, but they can be set to anything, in particular, the heap-size option (-Xmx).

When Tomcat is run as a service, CATALINA_OPTS is defined (and, therefore, modifiable) in /etc/systemd/system/tomcat.service.

First, how much memory does Tomcat have?

root@tirion:/home/russ# ps -ef | grep [t]omcat
tomcat    127078       1 71 15:12 ?        00:00:09 \
    /usr/lib/jvm/java-1.11.0-openjdk-amd64/bin/java \
    -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties \
    -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
    -Djava.awt.headless=true \
    -Djava.security.egd=file:/dev/./urandom \
    -Djdk.tls.ephemeralDHKeySize=2048 \
    -Djava.protocol.handler.pkgs=org.apache.catalina.webresources \
    -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 \
    -Xms512M -Xmx1024M -server -XX:+UseParallelGC \
    -Dignore.endorsed.dirs= \
    -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar \
    -Dcatalina.base=/opt/tomcat \
    -Dcatalina.home=/opt/tomcat \
    -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start

How to change this amount?

root@tirion:/etc/systemd/system# vim tomcat.service
    [Unit]
    Description=Apache Tomcat Web Application Container
    After=network.target

    [Service]
    Type=forking

    Environment=JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64
    Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
    Environment=CATALINA_HOME=/opt/tomcat
    Environment=CATALINA_BASE=/opt/tomcat
    Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
    Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

    ExecStart=/opt/tomcat/bin/startup.sh
    ExecStop=/opt/tomcat/bin/shutdown.sh

    User=tomcat
    Group=tomcat
    UMask=0007
    RestartSec=10
    Restart=always

    [Install]
    WantedBy=multi-user.target

So, how would one write a simple HTTP client to talk to an application (servlet) running in a Tomcat container? Assuming there's something easy to GET, do this first:

package com.windofkeltia.client;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

import static java.util.Objects.nonNull;

/**
 * How to get back the "I'm up and good" from out Tomcat-based application.
 * @author Russell Bateman
 * @since July 2021
 */
public class GetRequest
{
  private static final int    PORT     = 8080;
  private static final String BASE_URL = "http://localhost:" + PORT;
  private static final String APP_NAME = "application-3.2.0";

  public static void main( String ... args ) throws IOException
  {
    // set up HTTP connection and request...
    URL               url        = new URL( BASE_URL + '/' + APP_NAME );
    HttpURLConnection connection = ( HttpURLConnection ) url.openConnection();
    connection.setRequestMethod( "GET" );
    connection.setRequestProperty( "Accept", "text/plain" );

    // fire off request, then read response...
    int            status = connection.getResponseCode();
    BufferedReader reader = new BufferedReader( new InputStreamReader( connection.getInputStream() ) );
    String         line;
    StringBuilder  content = new StringBuilder();

    while( nonNull( line = reader.readLine() ) )
      content.append( line ).append( '\n' );

    reader.close();

    System.out.println( content );
  }
}

...and assuming there's some reason to POST to it, do this too:

package com.windofkeltia.client;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;

import static java.util.Objects.nonNull;

/**
 * How to post an XML document to our application, get back a response and read it.
 * @author Russell Bateman
 * @since July 2021
 */
public class PostRequest
{
  private static final int    PORT      = 8080;
  private static final String BASE_URL  = "http://localhost:" + PORT;
  private static final String APP_NAME  = "application-3.2.0";
  private static final String TESTING   = "src/test/resources/fodder/";
  private static final String FILENAME  = "test-file.xml";
  private static final String IXML_PATH = TESTING + "client/" + FILENAME;

  public static void main( String ... args ) throws IOException
  {
    // set up HTTP connection and request...
    URL               url        = new URL( BASE_URL + '/' + APP_NAME );
    HttpURLConnection connection = ( HttpURLConnection ) url.openConnection();
    connection.setRequestMethod( "POST" );
    connection.setRequestProperty( "Content-Type", "application/xml" );
    connection.setRequestProperty( "Accept",       "application/xml" );
    connection.setDoOutput( true );   // (otherwise, can't write payload to output stream)

    // build and POST payload...
    try( BufferedReader reader       = new BufferedReader( new FileReader( IXML_PATH ) );
        OutputStream   outputStream = connection.getOutputStream() )
    {
      for( String line = reader.readLine(); nonNull( line ); line = reader.readLine() )
      {
        final String PAYLOAD = line + '\n';
        outputStream.write( PAYLOAD.getBytes() );
      }
    }
    catch( Exception e )
    {
      e.printStackTrace();
    }

    // fire off request, then read response...
    int            status = connection.getResponseCode();
    BufferedReader reader = new BufferedReader( new InputStreamReader( connection.getInputStream() ) );
    String         line;
    StringBuilder  content = new StringBuilder();

    while( nonNull( line = reader.readLine() ) )
      content.append( line ).append( '\n' );

    reader.close();

    System.out.println( content );
  }
}

Besides GET and POST above, other HTTP interactions (HEAD, PUT, PATCH, DELETE, etc.) are similarly coded.

I found these four problems inspecting catalina.out, Tomcat's log:

1. I see this everywhere inside catalina.out. What is a "Jasper scratchDir"?

   org.apache.jasper.EmbeddedServletOptions The scratchDir you specified is unusable
   

   Missing information: I don't know what subdirectory may not permit adequate privileges (likely to Tomcat, tomcat:tomcat). Solution: check ownership and privileges under /opt/tomcat. Assessment: this is probably not a very important problem and likely has nothing to do with what I'm chasing down. The source of the problem may be that Tomcat is being started and stopped by the wrong user. What is the right user? tomcat, but isn't it managed thus?

   # systemctl start|stop|restart|status tomcat
   

2. Looking harder at catlina.out, I think I see this just before the cron job bounces Tomcat:

   com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within \
       MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
   

   This may have to do with mdht-restlet missing a parameter name

   <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
   

   which I did add to fhir-server. It would be good to try that and test whether it solves the problem on Tomcat. It's not likely anything to do with what we're looking for today.
   
   No. JSON is written to catalina.out at some point, but not by any code associated with the Jersey framework. This parameter is not needed.

3. The servlet makes calls into the MDHT library to construct the CCDA output (CCD), this is seen frequently in catalina.out:

   org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application created a \
       ThreadLocal with key of type [java.lang.ThreadLocal] and a value of type [org.eclipse.ocl.ecore.QueryImpl] \
       but failed to remove it when the web application was stopped
   

   Research demonstrates that this is something to expect when Tomcat is shut down. As it seems to be associated in time with the cron job bouncing, this isn't likely a real problem, but only resources that the thread acquires as it makes its way through the MDHT library and returns thence without freeing them. The notification comes, as pointed out, when Tomcat is being shut down.

Researching those messages above we do not understand...

Recommended best practice for memory sizes are below. They are set up when Tomcat is installed as a service.

Just so you know, if you're running multiple applications (servlets) under Tomcat, ...

Here's how to adjust Tomcat's JVM memory. Assuming Tomcat is running as a service, you should find the following path/filename:

/etc/systemd/system/tomcat.service

Heap size

• 2Gb for test servers (where data size is known to be average to small)
• 4Gb for production (improves performance)
• the default (1Gb) may work for private use (your mileage may vary)

In that file (tomcat.service), which you can edit (then bounce Tomcat), you should find a line like this:

Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'

The above is the default value (for Tomcat 9). -Xms refers to the stack size (see discussion below); -Xmx to the heap size. The latter (heap size) is probably what you want to increase. The value shown above there is 1 gigabyte. I would suggest -Xmx4096M as best practice for a starting value.

To bounce Tomcat after changes:

# systemctl restart tomcat

Stack size

Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'

-Xms1024M seems a good maximum to respect, but -Xms512M is the default (Tomcat 9) and best starting value.

Too big a stack size and each thread spun up to tackle a request by the application(s) Tomcat is serving depleats overall memory faster as each thread is allocated more stack size. Therefore: be certain you really need as much stack as you're giving. You'll know a thread has run short of stack memory when a java.lang.StackOverflowError appears in catalina.out.

To bounce Tomcat after changes:

# systemctl restart tomcat

Not running Tomcat as a service?

The above is done inside subdirectory /opt/tomcat/bin (or wherever your Tomcat files are in the filesystem).

Create a new file, setenv.sh:

export CATALINA_OPTS="$CATALINA_OPTS -Xms512M"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx1024M"

Then, bounce Tomcat, which you are running "locally" with:

$ ./catalina.sh run

You will see it launch and, confirmed somewhere in the logging that comes out to stdout, the following:

.
.
.
INFO [main] VersionLoggerListener.log Command line argument: -Xms512m
INFO [main] VersionLoggerListener.log Command line argument: -Xmx2048m
.
.
.

If catalina.out is removed after Tomcat is stopped, a new catalina.out will simply be created once Tomcat is restarted.

...including removing rotated log files.

Note that there is a default configuration Tomcat employs in the case of no system logrotate configuration for Tomcat, but, if you don't like the default behavior, do as below:

1. (I would never touch /opt/tomcat/conf/logging.properties.)
2. Instead, go to /etc/logrotate.d.
3. Note that, at /etc/cron.daily, there is a logrotate file. This cron job triggers /etc/logrotate.conf which includes all the system-wide log-rotation configuration files in /etc/logrotate.d. This can be made to work for Tomcat also.
4. Therefore, add a new file (if missing), tomcat.
5. In this file, put the path to the Tomcat log, e.g.: /opt/tomcat/logs/catalina.out.
6. Then add lines like the following thereafter:

   /opt/tomcat/logs/catalina.out
   {
     create                         # creates new, empty log file
     notifempty                     # creates new log file only if current one not empty
     daily|weekly|monthly           # choose one or...
     size 1m                        # rotate when log reaches 1Mb
     compress                       # compress rotated log using gzip
     missingok                      # don't error out if log missing
     rotate 7                       # keep at most 7 rotated logs, or
     maxage <days>                  # remove rotated logs after days
   }