Jetty on a vServer that limits numproc and numtcpsock

If you have a vServer it is likely that it has certain restrictions. E.g. many vServers only allow a limited number of threads to be running. You can see these limits using this command:


# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
 10247230:  kmemsize                 29523869             41385984            106385865            117024451                    0
            lockedpages                     0                    0                  860                  860                    0
            privvmpages                955727              1516817              2097152              2097152                    0
            shmpages                    46762                56592               196608               196608                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       215                  512                  512                  512                  537
            physpages                 1043148              1048606              1048576              1048576                    0
            vmguarpages                     0                    0              1048576           2147483647                    0
            oomguarpages               601150               808660              1048576           2147483647                    0
            numtcpsock                    274                  581                 1800                 1800                    0
            numflock                        9                   12                  500                  500                    0
            numpty                          5                    6                  128                  128                    0
            numsiginfo                      0                   51                 1024                 1024                    0
            tcpsndbuf                 1129480              3406320              4942675              7056211                    0
            tcprcvbuf                  617208              4396200              4942675              7056211                    0
            othersockbuf                39304               758232               844366              1481926                 2936
            dgramrcvbuf                     0                13344               844366               844366                    0
            numothersock                   54                   77                 1800                 1800                    0
            dcachesize                8079194              8110080              7299072              8110080                    0
            numfile                      1889                 2470                10000                10000                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      42                   56                  128                  128                    0

The “numproc” value is the number of allowed threads. In this example the system has exceeded this limit 537 times, which causes problems that e.g. Nagios cannot create sockets or that Java throws OutOfMemory exceptions. Using the program htop you can see how many threads each program uses:

If you are using Jetty or another Java based program you will see a lot of “java” threads. To find out what these threads are actually doing you can use the jstack command using the PID of your main jetty process, e.g. “jstack 12345”. It will then display a stacktrace of all running threads:

# jstack 14843
2019-03-25 10:25:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM:

"Okio Watchdog" #178 daemon prio=5 os_prio=0 tid=0x00007f40a00d2800 nid=0x6de9 in Object.wait() [0x00007f40933e1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:460)
	at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:361)
	at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312)
	- locked <0x00000000e127d148> (a java.lang.Class for okio.AsyncTimeout)

"pool-22-thread-1" #75 prio=5 os_prio=0 tid=0x00007f40942c3000 nid=0x68bf waiting on condition [0x00007f4084900000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000e4860618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

...

If you have problems that numproc is often exceeded then you can now try to find the threads that shouldn’t be there. E.g. despite I had limited Jettys threadpool it used more and more threads. The reason was that one of my Servlets used a Timer which was only used once but the system didn’t release it after it wasn’t used any more. So more and more Timer threads were created. The solution was to use a ScheduledExecutorService instead as explained on StackOverflow.

Another limitation is often numtcpsock. On my server it is limited to 1800 connections. If that limit is exceeded one cannot connect to the server any more e.g. via ssh. By default jetty does not seem to limit the number of connections it accepts. To limit it you can add these lines to the start.ini file for Jetty 9.4:

jetty.ssl.acceptQueueSize=200
jetty.http.acceptQueueSize=200

--module=connectionlimit
jetty.connectionlimit.maxConnections=200

It limits the number of connections to 200. Without it my server accepted more and more connections without being able to process them in time. After a short time the numtcpsock limit was exceeded and the server was not usable any more. Even my existing ssh connection broke down and I could not connect any more. But with this limit of 200 connections everything seems to work fine.