Apache JMeter Notes

Russ Bateman
January 2013
last update:

Simple collection of notes begun that will be organized down the road if there's any need to do so.

In order to execute JMeter, I had to use real Sun Java and not OpenJDK. I modified the last line of bin/jmeter thus:

    /home/russ/dev/jdk1.6.0_38/bin/java $ARGS $JVM_ARGS -jar `dirname $0`/ApacheJMeter.jar "$@"

Miscellaneous notes

Limit the number of threads. The harder JMeter is forced to work (i.e.: more threads), the more inflated timing gets because threads are waiting to gain access to the CPU. See Best Practices.

The test plan. This consists of:

See an example of a test plan here.

Output. JMeter reports warnings and error to jmeter.log.

Test files. The JMeter load tests are simply XML files, so you can check them in to your version-control system.


Running JMeter—options

...passed to shell script bin/jmeter:

  • -H http://web-proxy.austin.acme.com
  • -P 8080
  • -u username
  • -a password

JMeter links

The Apache documentation isn't very good. It's poorly written with bad grammar and mechanics, and the tutorials really aren't. There's no "JMeter for Dummies" step-by-step, start off simple documentation at all. You're better off looking elsewhere, like here:

Other useful links...


My First JMeter Test

This is the short tutorial referred to in the Links section on this page. I don't illustrate this fully because those illustrations are at the link.

  1. Launch JMeter.
  2. Right-click Test Plan, do Add -> Threads (Users) Thread Group.

    This allows us to control the number of threads (and hence virtual users), the ramp-up period, the loop count (how many times each virtual user will execute a test script), and the schedule (start- and end times and days).

  3. Right-click Test Plan, do Add -> Config Element -> HTTP Request Defaults.
            Server Name or IP: localhost            Port Number: 8000
    

    Later, if we want to run against some other server and port number, this is where we change them.

  4. Right-click Thread Group,* do Add -> Sampler -> HTTP Request.
            Name: Stats Request                     Path: /stats/stats
    

    * The original tutorial had this as right-click Test Plan -> Add -> Sampler, but of course, that's impossible.

  5. Right-click HTTP Request (Status Request), do Add -> Assertions -> Response Assertion.

    Then name this response assertion appropriately (it's for testing a 200 Okay return), click Response Code and Matches, then add 200 as a Patterns to Test by click Add, then scrolling up in the field and typing "200".

    Now we need some way of visualing the results, both Graph Results and Summary Report.

  6. Right-click Thread Group, do Add -> Listener -> Graph Results.
  7. (bis) Do the same to get a Summary Report. Name the file it will produce summary-report.

    Run the test.

  8. Choose Run -> Start.

    This will yield an alert saying, "You should save your test plan before running." Yes, you want to do this.

    Note in the Summary Report pane that the test was execited 25 times, averaging 2ms per request with a minimum response time of 1ms and a maximum of 14ms. There were errors 100% of the time (at very least because there was nothing at localhost:8000/stats/stats).

    There is nothing of interest in the Graph Results pane because there was no server to hit.

  9. Choose Run -> Clear or Clear All after making changes (like increasing the number of threads to run) before restarting.

My own first JMeter test: pinging my ReST application

This is my first actual test I set up on my own. I ping my server application, a restlet.


Look-up: my first JMeter ReST test attempt

Even after setting up the header content, which JMeter supports, I never am able to penetrate Jersey from JMeter (see log which reports a 404. Conclusion: I'll have to checkout some special ReST sampler plug-ins for JMeter.

I thought specifying the header content would be all that's different:


Look-up: my second attempt

The purpose of these notes is to document failure and success. Here's how I achieved a working GET with notes on what better to do:

  1. The HTTP Defaults aren't added to with each HTTP Request as I dimly surmised. Instead, they're the default overwriten by the latter. So, I had to put the whole path in the GET.
  2. I can see a lot more of what's going wrong by adding a listener, the Simple Data Writer: right-click Thread Group (here, Look-up), do Add -> Listener -> Simple Data Writer. Click Configure and ensure what's noted in the image below. As for what comes out, it's ugly, but useful:
    ~/dev/apache-jmeter-2.7/test-plans $ cat ../bin/lookup.log
    <?xml version="1.0" encoding="UTF-8"?>
    <testResults version="1.2">
    <httpSample t="13" lt="13" ts="1358467601684" s="true" lb="GET" rc="200" rm="OK" tn="Look-up 1-1" dt="text" by="774">
      <assertionResult>
        <name>Status 200 Okay</name>
        <failure>false</failure>
        <error>false</error>
      </assertionResult>
      <responseData
        class="java.lang.String"&gt;{&quot;oid&quot;:&quot;000000000000000000000992&quot;, \
        &quot;identities&quot;:&quot;[email protected]&quot;, \
        &quot;partneroid&quot;:&quot;000000000000000000000222&quot;, \
        &quot;username&quot;:&quot;venkat-partnerx&quot;, \ &quot;firstname&quot;:&quot;Venkat&quot;, \
        &quot;lastname&quot;:&quot;Mynampati&quot;,&quot;fullname&quot;:&quot;Venkat Mynampati&quot;, \
        &quot;phone&quot;:&quot;510-987-9874&quot;,&quot;mobile&quot;:&quot;510-987-9874&quot;, \
        &quot;fax&quot;:&quot;510-987-9874&quot;,&quot;birthdate&quot;:&quot;1973-01-01T05:12:12-07:00&quot;, \
        &quot;language&quot;:&quot;en&quot;,&quot;country&quot;:&quot;US&quot;, \
        &quot;accountdata&quot;:{&quot;tosaccepted&quot;:&quot;true&quot;, \
        &quot;favoritecolor&quot;:&quot;white&quot;,&quot;menuscheme&quot;:&quot;tandy&quot;, \
        &quot;nationalanthem&quot;:&quot;String of Pearls&quot;,&quot;bestchoice&quot;:&quot;selection7&quot;, \
        &quot;favoritefriend&quot;:&quot;Levi Miller&quot;,&quot;bestbeach&quot;:&quot;sandy&quot;}, \
        &quot;loggedin&quot;:&quot;false&quot;,&quot;created&quot;:&quot;2013-01-16T11:36:06.110-07:00&quot;, \
        &quot;forgotten&quot;:&quot;false&quot;}\
        &lt;/responseData>
      <java.net.URL>http://localhost:8000/accountmgr/api/v1/[email protected]</java.net.URL>
    </httpSample>
    </testResults>
    
  3. The HTTP Header Manager is different too; I had messed up the first time.


JMeter run script (sample)
    #!/bin/sh
    echo "Starting tests on remote JMeter Service..."
    DIR=/usr/local/jmeter2/jmeter
    ${DIR}/bin/jmeter.sh -n -t test-plan.jmx -l logfile.jtl -R 192.168.0.103 -Gtest.hostname=192.168.0.110 \
            -Gtest.port=443 -Gtest.protocol=https -Gtest.threads=10 -Gtest.ramp=10 -Gtest.loop=1 -X

JMeter run script for Jenkins (sample)
    #!/bin/sh
    echo "Starting tests on remote JMeter Service..."
    DIR=/opt/jmeter/jmeter
    ${DIR}/bin/jmeter.sh -n -t jmeter/test-plan.jmx -l build/reports/jmeter/performance.jtl $REMOTE_JMETER \
            $HOST_VALS $THREADS $MISC -X

JMeter configuration (sample)
# jmeter - JMeter Performance Testing Server
#
# JMeter Server provides a remote connection point to do performance testing.

description     "JMeter Server"

start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 3 5
umask 022

env JAVA_HOME=/usr/java/jdk6
env JMETER_HOME=/opt/jmeter/jmeter
env OPTS="-H web-proxy.acme.com -P 8080"

script
  PATH="$JAVA_HOME/bin:$JMETER_HOME/bin:$PATH"
  echo "PATH=$PATH"
  exec start-stop-daemon --start --chdir ${JMETER_HOME} --exec ${JMETER_HOME}/bin/jmeter-server -- $OPTS
end script

Here are notes/steps to installing Jmeter for remote use
  1. Modify /etc/hosts and add the real IP address for the host. JMeter requires this at startup.
  2. Download JMeter (latest version) and unzip it.
  3. Move it to the location /opt/jmeter
  4. Create a symbolic link (ln -s <jmeter_version> jmeter). This will create a link so that JMeter can always be found at /opt/jmeter/jmeter. When JMeter needs to be updated, just download the new version and change the symbolic link. If all apps use the /opt/jmeter/jmeter, nothing should break.
  5. Copy jmeter.conf from jmeter to the new JMeter server at /etc/init.d. This will create an upstart script.
  6. Now start jmeter-server using the upstart script using sudo start jmeter.

user.properties (sample)

user.properties is a file that resides on the remote JMeter that allows each instance to use unique values that aren't supplied by the client startup script.

It must be placed on the remote server in the ${JMETER_HOME}/bin directory.

This allows us to specify unique client id so that multiple instances don't stomp on each other (ex: id.start). Each instance must have unique values, so don't just copy the properties file to every instance without modifying the values.

    # Sample user.properties file
    #
    ##   Licensed to the Apache Software Foundation (ASF) under one or more
    ##   contributor license agreements.  See the NOTICE file distributed with
    ##   this work for additional information regarding copyright ownership.
    ##   The ASF licenses this file to You under the Apache License, Version 2.0
    ##   (the "License"); you may not use this file except in compliance with
    ##   the License.  You may obtain a copy of the License at
    ##
    ##       http://www.apache.org/licenses/LICENSE-2.0
    ##
    ##   Unless required by applicable law or agreed to in writing, software
    ##   distributed under the License is distributed on an "AS IS" BASIS,
    ##   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    ##   See the License for the specific language governing permissions and
    ##   limitations under the License.

    #---------------------------------------------------------------------------
    # Classpath configuration
    #---------------------------------------------------------------------------
    #
    # List of paths (separated by ;) to search for additional JMeter extension classes
    # - for example new GUI elements and samplers
    # These are in addition to lib/ext. Do not use this for utility jars.
    #search_paths=/app1/lib;/app2/lib

    # Users can define additional classpath items by setting the property below
    # - for example, utility jars or JUnit test cases
    #
    # Use the default separator for the host version of Java
    # Paths with spaces may cause problems for the JVM
    #user.classpath=../classes;../jars/jar1.jar

    #---------------------------------------------------------------------------
    # Logging configuration
    #---------------------------------------------------------------------------
    #log_level.jorphan.reflect=DEBUG
    # Warning: enabling the next debug line causes javax.net.ssl.SSLException: Received fatal alert: unexpected_message
    # for certain sites when used with the default HTTP Sampler
    #log_level.jmeter.util.HttpSSLProtocolSocketFactory=DEBUG
    #log_level.jmeter.util.JsseSSLManager=DEBUG

    # Enable Proxy request debug
    #log_level.jmeter.protocol.http.proxy.HttpRequestHdr=DEBUG

    id.start=1
    id.max=9999

This said, it's possible to ignore user.properties and set up defaults as if the values were there. This is useful because you want to commit your test plans to your source-code repository and not have to worry about what is or isn't in user.properties (or other properties files outside your source base).

In the illustration below, id.start is defined such that it has a default value of 1. The fields in

    ${__partner( field1, field2, field3 )}

are:

  1. variable name
  2. value assigned to (don't know what this is)
  3. default value

(Put elsewhere in this document:)

The ${__P(var, value)} variable is defined usually on the command line or defaulted as shown above for variables HOSTNAME, PORT and PROTOCOL. The second field is the default used when the definition cannot be found.


Redefining what's already been defined...

In the following illustration, HTTP Header Manager defines a number of states and expectations. These are what must appear in the HTTP request header passed to each invocation in the test.

However, to kick off our test, we've got to draw an OAuth token from the OAuth service. We must not pass Authorization in the header because that's not recognized (or tolerated) by OAuth. Therefore, we redefine it not to exist simply by "defining" it as not containing anything. We do this separately and specifically in the HTTP Get OAuth Token step. We also redefine Content-Type for that step because OAuth requires application/x-www-form-urlencoded instead of application/json.


Practical advice...

When running against your local web application, you might not want to leave some of the real load test variables the same. For example, I didn't want to run a test that created 100,000 users, used SSL and the wrong port number on my local development host. So I changed these values as shown below to 10 users, port 8000 and HTTP instead of HTTPS.

But, don't save these changes unless you mean to.

Also, don't run your web application in Eclipse in debug mode unless you want to babysit it against stopping on a breakpoint. On the other hand, running in debug mode with a breakpoint set would be a good way to track down something that only happens under load, in a big, long test, etc.


Some very practical examples...

This shows creating test steps that draw on the defaults and definitions discussed in earlier sections.

Add a new action, i.e.: a new test step in a restlet, here, one called "Account Manager". We right-click Users here. Most any action or test step we would want to define would be a Sampler, here specifically, an HTTP Request.

We want to specify creating a new account. Under that action, as it's going to be a POST operation, we can define the request payload. Notice that we prescribe the URL (path) and the JSON payload. JMeter excels in setting up load testing; so we use a sort of variable, ${USER_ID}, already defined, that will be incremented each time a new account is created so that we can create any number of account names (identities). This was handled by the User Incrementer higher up. Here, the password for each account is also varied.

After the operation is conducted, we are interested in the result. So, we add a Response Assertion.

In the response, we expect a response payload and, inside that payload which we know to be JSON, will be an "oid". So, we tell JMeter that the response must contain the text "oid".

Next, we want the value of that oid because it's concocted live by the underlying database. We cannot predict its value. So, we extract it via a Regular Expression Extractor:

We configure the extracto to look in the body (of the JSON response payload), for the oid key-value pair and extract it. This involves knowledge of regular expressions: we match the pattern enclosed between parens and say to use (match) pattern number 1. The template is something I haven't grok'd yet: see the JMeter documentation. (Sorry.) Arbitrarily, it's decided to stuff this value into a variable named acctOid, since this is the oid of an account. This variable is herewith defined: it did not need to be defined higher up.

Last, we show how acctOid is used by creating HTTP Find Account. This is a GET operation. Right-click Users and add this new HTTP Request action filling it out as below. The URI syntax dictates that we can find an existing user (we want to find the one just added) by adding its oid to the command line. This is the oid we just saved.

Last, after all the messing around we could do, we want to remove the account created as part of the test. This is so that the next time we run JMeter, the application won't complain that the user already exists. For this application, this isn't usually a permitted operation, so accommodate must be made to use the superuser's entity key, something that will force it to do the delete anyway. Notice acctOid again in the command line.

Here's how to inject a change into the HTTP header: we add a new Config Element, HTTP Manager with the superuser's key. Everything from the original HTTP Manager is upheld as it's inherited, but this one aspect is changed by us and only for the delete operation.

Here's the HTTP Header Manager change (the entitykey):


Getting results...

This is where the View Results Tree and Summary Report come in. You need them so you can see what is going on or happened.


Illustration of JMeter in Jenkins...

This illustrates how JMeter is set up in Jenkins for an application named accountmgr. This is only for the purpose of illstrating what needs to be or can be set up in Jenkins configuration.


Things not to do with JMeter scripts...
  1. Don't make the script too heavy. Reduce the number of validations, logic, response parses, debuggers, log-in information that weigh down the script.
  2. Consider real-world use of what you're testing. Will users have cookies that need to be erased? Do you need to parameterize the headers to make the script more flexible? What response assertions will be needed?
  3. Don't use absolute filesystem paths, so the script can be run on other hosts without modification. Gather filenames only; make the paths the object needs either relative or configurable.
  4. Don't leave URLs that don't really matter or are less readable, stuff from Google Analytics, plug-ins, Windows Update, etc.