SPF problem when forwarding emails

Forwarding emails can be a problem when the SPF (Sender Policy Framework) entry of the sender’s domain is set to “-all”. SPF is an email validation system designed to prevent spam and phishing by verifying that an incoming message was sent from an authorized server.

When an SPF record is set to “-all”, it indicates that no other servers are authorized to send email on behalf of the domain. This means that if an email is forwarded from an unauthorized server to another server, the receiving server may reject the message as a potential spam or phishing attempt.

E.g. if someone sends you an email from example.com and you want to automatically forward your emails from your domain myemailaddress.com to e.g. your GMail account, then this won’t work, because GMail won’t accept the email, because myemailaddress.com is not allowed to send “@example.com” emails. Only the mailservers of example.com are allowed to send such emails.

A solution can be to use an IMAP forwarder, i.e. a program that connects to both IMAP servers (myemailaddress.com and GMail in that example) and automatically sends all emails that appear on myemailaddress.com to GMail).

A free open source solution is the project IMAPTransfer on GitHub.

It is written in Kotlin and can watch the source server for changes and immediately write these new emails into the target IMAP server. Or it can write the new emails (or all emails to create a backup or an archive) into local .eml files . It can even restore these .eml files into another IMAP server.

Switching from altool to notarytool

Apple has deprecated altool which is used to sign programs so that they can be run on other people’s computers. They are signed with a developer’s key and “virus checked”.

Instead of using altool it will soon become necessary to use notarytool. Switching from altool to notarytool is simple. If you have used this before:

xcrun altool --notarize-app --primary-bundle-id com.your.app --username yourmail@yourmail.com --password yourpassword --file YourFile.dmg

You can now just use this command

xcrun notarytool submit YourFile.dmg --apple-id yourmail@yourmail.com --team-id YOURTEAMID --password yourpassword --wait

There is also a way to save the password in a keychain profile using the “store-credentials” command. However the build script cannot access the keychain profile while the device is locked, i.e. you could only run the build script while using the computer.

Fixing curl’s Let’s Encrypt problem on Linux

One of Let’s Encrypt’s root certificates (“DST Root CA X3”) has expired on September 30th, 3021. Now a new root certificate (“ISRG Root X1”) is used. Let’s Encrypt’s Intermediate certificate “Let’s Encrypt R3” was signed by both root certificates. Older versions of curl (7.52) cannot handle this correctly and think that the R3 certificate is no longer valid because its root certificate has expired. However only one of the two root certificates has expired, the other is still valid and thus R3 is valid, too. You can find a diagram of the certificates here: https://letsencrypt.org/certificates/

When you try to use curl 7.52 it can look like this:

# curl --head https://blog.dgunia.de
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html

So one option is to update curl to a newer version, e.g. 7.64 or 7.74, then it works fine. Another option is to remove the expired root certificate (“DST Root CA X3”) from the Linux computer on which you want to use curl.

To remove the certificate, just edit the file /etc/ca-certificates.conf and disable the DST Root CA X3 certificate by writing an exclamation mark in front of it:

!mozilla/DST_Root_CA_X3.crt

Then run update-ca-certificates to read the ca-certificates.conf file and update the system’s certificates. Afterward curl should work fine.

Update: When I compiled the new version of curl I also had to compile a new version of OpenSSL. It seems that the real problem is in OpenSSL 1.0, not in curl. So it seems to be sufficient to update OpenSSL 1.0 to at least OpenSSL 1.1 or to remove the expired certificate. You can now find a blog post that explains this here:

https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Making a Java program available as default email handler on Windows

This page will explain how to register a JavaFX maven program that uses the javafx-maven-plugin as a default email handler on Windows. I.e. users can see that program as an option when selecting the default email program.

To customize the installation process of Inno Setup and WiX toolset you need the default configuration files Java uses. In JDK 8 the template.iss and template.wxs files can be found under com/oracle/tools/packager/windows/template.iss in the jar %JAVA_HOME%\lib\ant-javafx.jar ( https://stackoverflow.com/questions/31862568/how-can-i-customize-the-inno-template-iss-file-to-be-used-in-a-javafx-8-build ). You have to copy these files into src/main/deploy/package/windows and rename them to yourapp.iss and yourapp.wxs . Then you can add additional entries to create registry entries to them.

For InnoDB you have to add these lines:

[Registry]

Root: HKCU; Subkey: "SOFTWARE\Classes\YourApp.MailTo"; Flags: uninsdeletekey; ValueType: string; ValueName: ""; ValueData: "URL:MailTo Protocol"
Root: HKCU; Subkey: "SOFTWARE\Classes\YourApp.MailTo"; Flags: uninsdeletekey; ValueType: string; ValueName: "URL Protocol"; ValueData: ""
Root: HKCU; Subkey: "SOFTWARE\Classes\YourApp.MailTo\shell"; Flags: uninsdeletekey
Root: HKCU; Subkey: "SOFTWARE\Classes\YourApp.MailTo\shell\open"; Flags: uninsdeletekey
Root: HKCU; Subkey: "SOFTWARE\Classes\YourApp.MailTo\shell\open\command"; Flags: uninsdeletekey; ValueType: string; ValueName: ""; ValueData: "{app}\YourApp.exe ""%1"" ""%2"" ""%3"" ""%4"" ""%5"" ""%6"" ""%7"" ""%8"" ""%9"""

Root: HKCU; Subkey: "SOFTWARE\Classes\Applications\YourApp"; Flags: uninsdeletekey; ValueType: string; ValueName: ""; ValueData: "URL:MailTo Protocol"
Root: HKCU; Subkey: "SOFTWARE\Classes\Applications\YourApp\DefaultIcon"; Flags: uninsdeletekey; ValueType: string; ValueName: "URL Protocol"; ValueData: "{app}\YourApp.exe"
Root: HKCU; Subkey: "SOFTWARE\Classes\Applications\YourApp\FriendlyAppName"; Flags: uninsdeletekey; ValueType: string; ValueName: ""; ValueData: "YourApp"
Root: HKCU; Subkey: "SOFTWARE\Classes\Applications\YourApp\shell"; Flags: uninsdeletekey
Root: HKCU; Subkey: "SOFTWARE\Classes\Applications\YourApp\shell\open"; Flags: uninsdeletekey
Root: HKCU; Subkey: "SOFTWARE\Classes\Applications\YourApp\shell\open\command"; Flags: uninsdeletekey; ValueType: string; ValueName: ""; ValueData: "{app}\YourApp.exe ""%1"" ""%2"" ""%3"" ""%4"" ""%5"" ""%6"" ""%7"" ""%8"" ""%9"""

Root: HKCU; Subkey: "SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\YourApp"; Flags: uninsdeletekey; ValueType: string; ValueName: ""; ValueData: "{app}\YourApp.exe"
Root: HKCU; Subkey: "SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\YourApp"; Flags: uninsdeletekey; ValueType: string; ValueName: "SupportedProtocols"; ValueData: "mailto"

Root: HKCU; Subkey: "SOFTWARE\Clients\StartMenuInternet\YourApp\Capabilities\UrlAssociations"; Flags: uninsdeletekey; ValueType: string; ValueName: "mailto"; ValueData: "YourApp.MailTo"

Root: HKCU; Subkey: "SOFTWARE\RegisteredApplications"; Flags: uninsdeletekey; ValueType: string; ValueName: "YourApp"; ValueData: "SOFTWARE\Clients\StartMenuInternet\YourApp\Capabilities"

They will first register a new class “YourApp.MailTo” that can start your program with command line parameters. Then it will register your program in the “Applications” folder and give it a “FriendlyAppName” so that the system knows how to display it. Afterward it will register your program in the “App Paths” and tell the system that it supports the “mailto” protocol. It creates an UrlAssociation from “mailto” to your “YourApp.MailTo” class and registers it in the “RegisteredApplications” under the name of your program.

You can do the same in the wxs file for creating MSI files. Just add these lines to the component “CleanupMainApplicationFolder”:

<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\YourApp.MailTo">
<RegistryValue Type="string" Value="URL:MailTo Protocol"/>
<RegistryValue Type="string" Name="URL Protocol" Value=""/>
</RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\YourApp.MailTo\DefaultIcon">
<RegistryValue Type="string" Name="URL Protocol" Value="[APPLICATIONFOLDER]YourApp.exe"/>
</RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\YourApp.MailTo\shell"></RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\YourApp.MailTo\shell\open"></RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\YourApp.MailTo\shell\open\command">
<RegistryValue Type="string" Value="[APPLICATIONFOLDER]YourApp.exe &quot;%1&quot; &quot;%2&quot; &quot;%3&quot; &quot;%4&quot; &quot;%5&quot; &quot;%6&quot; &quot;%7&quot; &quot;%8&quot; &quot;%9&quot;"/>
</RegistryKey>

<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\Applications\YourApp">
<RegistryValue Type="string" Value="URL:MailTo Protocol"/>
<RegistryValue Type="string" Name="URL Protocol" Value=""/>
</RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\Applications\YourApp\DefaultIcon">
<RegistryValue Type="string" Name="URL Protocol" Value="[APPLICATIONFOLDER]YourApp.exe"/>
</RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\Applications\YourApp\FriendlyAppName">
<RegistryValue Type="string" Value="YourApp"/>
</RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\Applications\YourApp\shell"></RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\Applications\YourApp\shell\open"></RegistryKey>
<RegistryKey Root="HKCU" Key="SOFTWARE\Classes\Applications\YourApp\shell\open\command">
<RegistryValue Type="string" Value="[APPLICATIONFOLDER]YourApp.exe &quot;%1&quot; &quot;%2&quot; &quot;%3&quot; &quot;%4&quot; &quot;%5&quot; &quot;%6&quot; &quot;%7&quot; &quot;%8&quot; &quot;%9&quot;"/>
</RegistryKey>

<RegistryKey Root="HKCU" Key="SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\YourApp">
<RegistryValue Type="string" Value="[APPLICATIONFOLDER]YourApp.exe &quot;%1&quot; &quot;%2&quot; &quot;%3&quot; &quot;%4&quot; &quot;%5&quot; &quot;%6&quot; &quot;%7&quot; &quot;%8&quot; &quot;%9&quot;"/>
<RegistryValue Type="string" Name="SupportedProtocols" Value="mailto"/>
</RegistryKey>

<RegistryKey Root="HKCU" Key="SOFTWARE\Clients\StartMenuInternet\YourApp\Capabilities\UrlAssociations">
<RegistryValue Type="string" Name="mailto" Value="YourApp.MailTo"/>
</RegistryKey>

<RegistryKey Root="HKCU" Key="SOFTWARE\RegisteredApplications">
<RegistryValue Type="string" Name="YourApp" Value="SOFTWARE\Clients\StartMenuInternet\YourApp\Capabilities"/>
</RegistryKey>

Accessing the Android Simulator on a remote computer via VNC

If you are using VNC to access a remote Linux computer and you are trying to run the Android Emulator you might see either a blank window or a distorted window:

Blank window, probably due to the “Hardware” graphics driver.
Distorted window, probably due to a wrong display depth.


To fix it you have to select the “Software” graphics driver when creating the emulator:

Additionally when running your VNC server you have to run it in 24 bit mode:

vncserver :2 -depth 24

Then it should work fine:

Signed macOS programs with Java 14

Since February 2020 Apple requires all programs to be signed, hardened and notarized so that the Gatekeeper on macOS Catalina allows them to be run. Before it was only necessary to sign them so Java 8 could still be used. Now they have to be hardened, with requires XCode 10 and Java 8 cannot be compiled with XCode 10 yet so a newer Java version has to be used. Java 14 contains the “jpackage” program to create a native signed app that can fulfill these requirements (with some additional work).

First you have to build your program, e.g. with maven

mvn package

Then you have to create an app image using jpackage e.g. this way:

$JAVA_HOME/bin/jpackage -n MyApp --input target --main-jar MyApp-1.0.jar --main-class com.myapp.MyApp --module-path libfx --add-modules javafx.controls,javafx.fxml,javafx.web,javafx.swing,javafx.media --icon src/main/deploy/package/macosx/MyApp.icns --type app-image --dest appimageoutput --java-options "-Xmx1024m"

Afterward you have to sign all jar files and dylib files in the appimageoutput directory. My “SignPackage.jar” simply searches for all dylib and jar files in the given directory (also dylib files inside jar files) and signs them with “codesign –timestamp –options runtime –entitlements … — deep -vvv -f –sign “Developer (XXX)” file”:

java -jar SignPackage.jar -d appimageoutput -t -r -k "Developer ID Application: John Public (XXXXXXXXXX)" -e "src/main/deploy/package/macosx/MyApp.entitlements"

codesign --timestamp --entitlements src/main/deploy/package/macosx/MyApp.entitlements --options runtime --deep -vvv -f --sign "Developer ID Application: John Public (XXXXXXXXXX)" appimageoutput/MyApp.app/Contents/MacOS/*

codesign --timestamp --entitlements src/main/deploy/package/macosx/MyApp.entitlements --options runtime --deep -vvv -f --sign "Developer ID Application: John Public (XXXXXXXXXX)" appimageoutput/MyApp.app

Then you can create a DMG file:

$JAVA_HOME/bin/jpackage -n MyApp --mac-package-identifier com.myapp --mac-package-name MyApp --mac-sign --mac-signing-key-user-name "John Public (XXXXXXXXXX)" --app-image appimageoutput

Sign the DMG file:

codesign --timestamp --entitlements src/main/deploy/package/macosx/MyApp.entitlements --options runtime --deep -vvv -f --sign "Developer ID Application: John Public (XXXXXXXXXX)" MyApp-1.0.dmg

And notarize it:

xcrun altool --notarize-app --primary-bundle-id com.myapp --username john@public.com --password mypassword --file MyApp-1.0.dmg

The entitlements file should probably at least contain these lines:

<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-executable-page-protection</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>

A complete “MyApp.entitlements” file could look like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.app-sandbox</key>
<false/>
<key>com.apple.security.network.server</key>
<true/>
<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.files.user-selected.read-write</key>
<true/>
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.disable-executable-page-protection</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
</dict>
</plist>

Slow MacBook/Notebook?

If you think your MacBook or notebook is slower than when you bought it you might be right. If you have an Intel CPU a dirty fan can make your computer slower because the CPU gets too hot and thus the CPU frequency is reduced. You can examine it by installing the Intel Power Gadget. This is how it looks if the fan cannot cool the CPU enough so that it stays at 100°C:

What you can see here is that if the CPU is used much (“Utilization”) the temperature reaches 100°C. And to decrease the temperature the CPU frequency is reduced, making the CPU slower. In this case it had only 1,7 Ghz in the end.

After cleaning the fans of the MacBook the cooling worked better and it became much faster again.

Moving a website from one provider to another without interruption

When moving a website from one provider to another you probably want to do this without any downtime, i.e. your users should not notice that the provider has changed. Here are a few tips for this process:

Moving static pages (html, css, …)

For moving static pages you can just copy them from your old webspace onto your new webspace, either using a program on your computer or directly. My old provider and my new provider offered ssh access. So I could just use rsync to copy the data from the old server to the new. I had to run a command like this on the new server:

rsync -rtv --links oldusername@oldserver.com:* www/

Later I could run it again to incrementally copy only changed files.

Creating an SSL certificate for the not-yet connected domain

Let’s say you are moving https://www.mydomain.com from one provider to another. You will need to setup an SSL certificate on the new provider before moving the domain there. Because otherwise there would be a downtime between moving the domain and setting up the SSL certificate. If your provider allows to manually enter a certificate and you cannot download the certificate from your old provider, you can use the “manual” mode of “Let’s encrypt” to create the necessary files. First you have to install Certbot. Then you can create a certificate like this:

mkdir certificate
cd certificate

mkdir logs
mkdir etc
mkdir work

certbot certonly -a manual -i apache -d www.mydomain.com --logs-dir logs --config-dir etc --work-dir work

During the process you will have to create a directory .well-known/acme-challenge on your old server and copy a file with a certain content into it. If you have ssh access it can look like this:

mkdir .well-known
cd .well-known
mkdir acme-challenge
cd acme-challenge
cat > thechallengefilename

Then copy&paste the content of the challenge file into the ssh shell and press CTRL-D and then CTRL-C. Afterward you can continue with certbot.

Then you should have four files in etc/live/www.mydomain.com:

cert.pem    chain.pem   fullchain.pem   privkey.pem

You have to copy the contents of these files onto your new provider’s server as your new SSL certificate. It depends on your provider where you have to put it. E.g. on my provider there was a “manual” mode that allowed to copy cert.pem into the CRT field, privkey.pem into the PrivateKey field and chain.pem into the CAT field. Afterward the SSL certificate was successfully installed.

Moving PHP files and databases

If you have also dynamic content, e.g. PHP files and databases, you have the problem that during the transfer of your domain you cannot know which server a user will use. Because the DNS entry of your domain is saved on multiple DNS servers and it can take 24 hours until they all point to the same server, i.e. to your new server. During that time both servers will be used. However that could mean that some data would be saved into your old database and some into your new and you would have to merge both. To prevent this you can just forward all requests from your old server to your new server using a PHP script like this (from StackOverflow):


&lt;?php

// https://stackoverflow.com/questions/22437548/php-how-to-redirect-forward-http-request-with-header-and-body

error_reporting(E_ALL);
ini_set('display_errors', 1);

/* Set it true for debugging. */
$logHeaders = FALSE;

/* Site to forward requests to.  */
$site = 'https://www.newdomain.com/';

/* Domains to use when rewriting some headers. */
$remoteDomain = 'www.newdomain.com';
$proxyDomain = 'www.olddomain.com';

$request = $_SERVER&#91;'REQUEST_URI'];

$ch = curl_init();

/* If there was a POST request, then forward that as well.*/
if ($_SERVER&#91;'REQUEST_METHOD'] == 'POST')
{
    curl_setopt($ch, CURLOPT_POST, TRUE);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
}
curl_setopt($ch, CURLOPT_URL, $site . $request);
curl_setopt($ch, CURLOPT_HEADER, TRUE);

$headers = getallheaders();

/* Translate some headers to make the remote party think we actually browsing that site. */
$extraHeaders = array();
if (isset($headers&#91;'Referer']))
{
    $extraHeaders&#91;] = 'Referer: '. str_replace($proxyDomain, $remoteDomain, $headers&#91;'Referer']);
}
if (isset($headers&#91;'Origin']))
{
    $extraHeaders&#91;] = 'Origin: '. str_replace($proxyDomain, $remoteDomain, $headers&#91;'Origin']);
}

/* Forward cookie as it came.  */
curl_setopt($ch, CURLOPT_HTTPHEADER, $extraHeaders);
if (isset($headers&#91;'Cookie']))
{
    curl_setopt($ch, CURLOPT_COOKIE, $headers&#91;'Cookie']);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);

if ($logHeaders)
{
    $f = fopen("headers.txt", "a");
    curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
    curl_setopt($ch, CURLOPT_STDERR, $f);
}

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);

$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$headers = substr($response, 0, $header_size);
$body = substr($response, $header_size);

$headerArray = explode(PHP_EOL, $headers);

/* Process response headers. */
foreach($headerArray as $header)
{
    $colonPos = strpos($header, ':');
    if ($colonPos !== FALSE)
    {
        $headerName = substr($header, 0, $colonPos);

        /* Ignore content headers, let the webserver decide how to deal with the content. */
        if (trim($headerName) == 'Content-Encoding') continue;
        if (trim($headerName) == 'Content-Length') continue;
        if (trim($headerName) == 'Transfer-Encoding') continue;
        if (trim($headerName) == 'Location') continue;
        /* -- */
        /* Change cookie domain for the proxy */
        if (trim($headerName) == 'Set-Cookie')
        {
            $header = str_replace('domain='.$remoteDomain, 'domain='.$proxyDomain, $header);
        }
        /* -- */

    }
    header($header, FALSE);
}

echo $body;

if ($logHeaders)
{
    fclose($f);
}
curl_close($ch);

?&gt;

And you have to forward all requests to that file by creating a .htaccess file with the following content in the root directory of your old server’s webspace:

RewriteEngine On
RewriteRule .* proxy.php

The problem here is that you need a domain with a different name to forward the requests to. Or you could maybe enter your new server’s IP. I just created another domain (a subdomain of an existing domain that I had already transferred), e.g. transfer.myotherdomain.com and let it host the same files as my main domain. I.e. on the old server I still had the domain www.mydomain.com but now with the .htaccess and proxy.php files. On the new server I had www.mydomain.com and transfer.myotherdomain.com which both pointed to the same web directory and had the same contents.

This way the old www.mydomain.com could forward all requests to the new server using transfer.myotherdomain.com because “www.mydomain.com” was not accessible on the new server yet. When I then transferred the www.mydomain.com domain from the old provider to the new provider the www.mydomain.com on the new server started to receive requests and replaced the old server.

Before uploading the .htaccess file you have to copy (export/import) your databases from your old server onto your new server and enter the new database connections and passwords in your php files if necessary.

Moving email accounts

Just setup the same accounts on your new server that you had on your old server. Then use e.g. Thunderbird to connect to both servers and move the emails from one server to the other or just archive them on your computer.

Moving the domain name

When you move your domain name from one hoster to another you have to configure the nameserver on the target hoster to the same IPs as the original hoster first (until you are ready to make the switch). And when you have done that you should set the nameservers of your domain in the original hoster’s web interface already to the nameservers of your new hoster and wait 24 hours. Because what can happen is when you actually move your domain from one hoster to another that your original hoster will immediately remove your entries from their nameservers. So that clients that don’t know your new nameserver yet (because they are using an old cached response that says that your nameserver is the original nameserver) will ask the wrong nameserver and will get no answer. So you should do this:

  • When you are moving from hoster A to B then configure B’s nameserver in the same way as A’s nameserver (i.e. the same entries and IPs, just different nameservers).
  • Then open the web interface of hoster A and enter the nameservers of hoster B for your domain.
  • Wait 24 hours so that everyone knows the new nameservers.
  • Move your domain from hoster A to hoster B.

Finished

After all these steps you should have successfully moved your domain from one provider to another and the users shouldn’t have noticed it because there was no downtime.

Jetty on a vServer that limits numproc and numtcpsock

If you have a vServer it is likely that it has certain restrictions. E.g. many vServers only allow a limited number of threads to be running. You can see these limits using this command:


# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
 10247230:  kmemsize                 29523869             41385984            106385865            117024451                    0
            lockedpages                     0                    0                  860                  860                    0
            privvmpages                955727              1516817              2097152              2097152                    0
            shmpages                    46762                56592               196608               196608                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       215                  512                  512                  512                  537
            physpages                 1043148              1048606              1048576              1048576                    0
            vmguarpages                     0                    0              1048576           2147483647                    0
            oomguarpages               601150               808660              1048576           2147483647                    0
            numtcpsock                    274                  581                 1800                 1800                    0
            numflock                        9                   12                  500                  500                    0
            numpty                          5                    6                  128                  128                    0
            numsiginfo                      0                   51                 1024                 1024                    0
            tcpsndbuf                 1129480              3406320              4942675              7056211                    0
            tcprcvbuf                  617208              4396200              4942675              7056211                    0
            othersockbuf                39304               758232               844366              1481926                 2936
            dgramrcvbuf                     0                13344               844366               844366                    0
            numothersock                   54                   77                 1800                 1800                    0
            dcachesize                8079194              8110080              7299072              8110080                    0
            numfile                      1889                 2470                10000                10000                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      42                   56                  128                  128                    0

The “numproc” value is the number of allowed threads. In this example the system has exceeded this limit 537 times, which causes problems that e.g. Nagios cannot create sockets or that Java throws OutOfMemory exceptions. Using the program htop you can see how many threads each program uses:

If you are using Jetty or another Java based program you will see a lot of “java” threads. To find out what these threads are actually doing you can use the jstack command using the PID of your main jetty process, e.g. “jstack 12345”. It will then display a stacktrace of all running threads:

# jstack 14843
2019-03-25 10:25:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM:

"Okio Watchdog" #178 daemon prio=5 os_prio=0 tid=0x00007f40a00d2800 nid=0x6de9 in Object.wait() [0x00007f40933e1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:460)
	at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:361)
	at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312)
	- locked <0x00000000e127d148> (a java.lang.Class for okio.AsyncTimeout)

"pool-22-thread-1" #75 prio=5 os_prio=0 tid=0x00007f40942c3000 nid=0x68bf waiting on condition [0x00007f4084900000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000e4860618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

...

If you have problems that numproc is often exceeded then you can now try to find the threads that shouldn’t be there. E.g. despite I had limited Jettys threadpool it used more and more threads. The reason was that one of my Servlets used a Timer which was only used once but the system didn’t release it after it wasn’t used any more. So more and more Timer threads were created. The solution was to use a ScheduledExecutorService instead as explained on StackOverflow.

Another limitation is often numtcpsock. On my server it is limited to 1800 connections. If that limit is exceeded one cannot connect to the server any more e.g. via ssh. By default jetty does not seem to limit the number of connections it accepts. To limit it you can add these lines to the start.ini file for Jetty 9.4:

jetty.ssl.acceptQueueSize=200
jetty.http.acceptQueueSize=200

--module=connectionlimit
jetty.connectionlimit.maxConnections=200

It limits the number of connections to 200. Without it my server accepted more and more connections without being able to process them in time. After a short time the numtcpsock limit was exceeded and the server was not usable any more. Even my existing ssh connection broke down and I could not connect any more. But with this limit of 200 connections everything seems to work fine.