Moving a website from one provider to another without interruption

When moving a website from one provider to another you probably want to do this without any downtime, i.e. your users should not notice that the provider has changed. Here are a few tips for this process:

Moving static pages (html, css, …)

For moving static pages you can just copy them from your old webspace onto your new webspace, either using a program on your computer or directly. My old provider and my new provider offered ssh access. So I could just use rsync to copy the data from the old server to the new. I had to run a command like this on the new server:

rsync -rtv --links oldusername@oldserver.com:* www/

Later I could run it again to incrementally copy only changed files.

Creating an SSL certificate for the not-yet connected domain

Let’s say you are moving https://www.mydomain.com from one provider to another. You will need to setup an SSL certificate on the new provider before moving the domain there. Because otherwise there would be a downtime between moving the domain and setting up the SSL certificate. If your provider allows to manually enter a certificate and you cannot download the certificate from your old provider, you can use the “manual” mode of “Let’s encrypt” to create the necessary files. First you have to install Certbot. Then you can create a certificate like this:

mkdir certificate
cd certificate

mkdir logs
mkdir etc
mkdir work

certbot certonly -a manual -i apache -d www.mydomain.com --logs-dir logs --config-dir etc --work-dir work

During the process you will have to create a directory .well-known/acme-challenge on your old server and copy a file with a certain content into it. If you have ssh access it can look like this:

mkdir .well-known
cd .well-known
mkdir acme-challenge
cd acme-challenge
cat > thechallengefilename

Then copy&paste the content of the challenge file into the ssh shell and press CTRL-D and then CTRL-C. Afterward you can continue with certbot.

Then you should have four files in etc/live/www.mydomain.com:

cert.pem    chain.pem   fullchain.pem   privkey.pem

You have to copy the contents of these files onto your new provider’s server as your new SSL certificate. It depends on your provider where you have to put it. E.g. on my provider there was a “manual” mode that allowed to copy cert.pem into the CRT field, privkey.pem into the PrivateKey field and chain.pem into the CAT field. Afterward the SSL certificate was successfully installed.

Moving PHP files and databases

If you have also dynamic content, e.g. PHP files and databases, you have the problem that during the transfer of your domain you cannot know which server a user will use. Because the DNS entry of your domain is saved on multiple DNS servers and it can take 24 hours until they all point to the same server, i.e. to your new server. During that time both servers will be used. However that could mean that some data would be saved into your old database and some into your new and you would have to merge both. To prevent this you can just forward all requests from your old server to your new server using a PHP script like this (from StackOverflow):


<?php

// https://stackoverflow.com/questions/22437548/php-how-to-redirect-forward-http-request-with-header-and-body

error_reporting(E_ALL);
ini_set('display_errors', 1);

/* Set it true for debugging. */
$logHeaders = FALSE;

/* Site to forward requests to.  */
$site = 'https://www.newdomain.com/';

/* Domains to use when rewriting some headers. */
$remoteDomain = 'www.newdomain.com';
$proxyDomain = 'www.olddomain.com';

$request = $_SERVER['REQUEST_URI'];

$ch = curl_init();

/* If there was a POST request, then forward that as well.*/
if ($_SERVER['REQUEST_METHOD'] == 'POST')
{
    curl_setopt($ch, CURLOPT_POST, TRUE);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
}
curl_setopt($ch, CURLOPT_URL, $site . $request);
curl_setopt($ch, CURLOPT_HEADER, TRUE);

$headers = getallheaders();

/* Translate some headers to make the remote party think we actually browsing that site. */
$extraHeaders = array();
if (isset($headers['Referer']))
{
    $extraHeaders[] = 'Referer: '. str_replace($proxyDomain, $remoteDomain, $headers['Referer']);
}
if (isset($headers['Origin']))
{
    $extraHeaders[] = 'Origin: '. str_replace($proxyDomain, $remoteDomain, $headers['Origin']);
}

/* Forward cookie as it came.  */
curl_setopt($ch, CURLOPT_HTTPHEADER, $extraHeaders);
if (isset($headers['Cookie']))
{
    curl_setopt($ch, CURLOPT_COOKIE, $headers['Cookie']);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);

if ($logHeaders)
{
    $f = fopen("headers.txt", "a");
    curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
    curl_setopt($ch, CURLOPT_STDERR, $f);
}

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);

$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$headers = substr($response, 0, $header_size);
$body = substr($response, $header_size);

$headerArray = explode(PHP_EOL, $headers);

/* Process response headers. */
foreach($headerArray as $header)
{
    $colonPos = strpos($header, ':');
    if ($colonPos !== FALSE)
    {
        $headerName = substr($header, 0, $colonPos);

        /* Ignore content headers, let the webserver decide how to deal with the content. */
        if (trim($headerName) == 'Content-Encoding') continue;
        if (trim($headerName) == 'Content-Length') continue;
        if (trim($headerName) == 'Transfer-Encoding') continue;
        if (trim($headerName) == 'Location') continue;
        /* -- */
        /* Change cookie domain for the proxy */
        if (trim($headerName) == 'Set-Cookie')
        {
            $header = str_replace('domain='.$remoteDomain, 'domain='.$proxyDomain, $header);
        }
        /* -- */

    }
    header($header, FALSE);
}

echo $body;

if ($logHeaders)
{
    fclose($f);
}
curl_close($ch);

?>

And you have to forward all requests to that file by creating a .htaccess file with the following content in the root directory of your old server’s webspace:

RewriteEngine On
RewriteRule .* proxy.php

The problem here is that you need a domain with a different name to forward the requests to. Or you could maybe enter your new server’s IP. I just created another domain (a subdomain of an existing domain that I had already transferred), e.g. transfer.myotherdomain.com and let it host the same files as my main domain. I.e. on the old server I still had the domain www.mydomain.com but now with the .htaccess and proxy.php files. On the new server I had www.mydomain.com and transfer.myotherdomain.com which both pointed to the same web directory and had the same contents.

This way the old www.mydomain.com could forward all requests to the new server using transfer.myotherdomain.com because “www.mydomain.com” was not accessible on the new server yet. When I then transferred the www.mydomain.com domain from the old provider to the new provider the www.mydomain.com on the new server started to receive requests and replaced the old server.

Before uploading the .htaccess file you have to copy (export/import) your databases from your old server onto your new server and enter the new database connections and passwords in your php files if necessary.

Moving email accounts

Just setup the same accounts on your new server that you had on your old server. Then use e.g. Thunderbird to connect to both servers and move the emails from one server to the other or just archive them on your computer.

Finished

After all these steps you should have successfully moved your domain from one provider to another and the users shouldn’t have noticed it because there was no downtime.

Jetty on a vServer that limits numproc and numtcpsock

If you have a vServer it is likely that it has certain restrictions. E.g. many vServers only allow a limited number of threads to be running. You can see these limits using this command:


# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
 10247230:  kmemsize                 29523869             41385984            106385865            117024451                    0
            lockedpages                     0                    0                  860                  860                    0
            privvmpages                955727              1516817              2097152              2097152                    0
            shmpages                    46762                56592               196608               196608                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       215                  512                  512                  512                  537
            physpages                 1043148              1048606              1048576              1048576                    0
            vmguarpages                     0                    0              1048576           2147483647                    0
            oomguarpages               601150               808660              1048576           2147483647                    0
            numtcpsock                    274                  581                 1800                 1800                    0
            numflock                        9                   12                  500                  500                    0
            numpty                          5                    6                  128                  128                    0
            numsiginfo                      0                   51                 1024                 1024                    0
            tcpsndbuf                 1129480              3406320              4942675              7056211                    0
            tcprcvbuf                  617208              4396200              4942675              7056211                    0
            othersockbuf                39304               758232               844366              1481926                 2936
            dgramrcvbuf                     0                13344               844366               844366                    0
            numothersock                   54                   77                 1800                 1800                    0
            dcachesize                8079194              8110080              7299072              8110080                    0
            numfile                      1889                 2470                10000                10000                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      42                   56                  128                  128                    0

The “numproc” value is the number of allowed threads. In this example the system has exceeded this limit 537 times, which causes problems that e.g. Nagios cannot create sockets or that Java throws OutOfMemory exceptions. Using the program htop you can see how many threads each program uses:

If you are using Jetty or another Java based program you will see a lot of “java” threads. To find out what these threads are actually doing you can use the jstack command using the PID of your main jetty process, e.g. “jstack 12345”. It will then display a stacktrace of all running threads:

# jstack 14843
2019-03-25 10:25:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM:

"Okio Watchdog" #178 daemon prio=5 os_prio=0 tid=0x00007f40a00d2800 nid=0x6de9 in Object.wait() [0x00007f40933e1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:460)
	at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:361)
	at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312)
	- locked <0x00000000e127d148> (a java.lang.Class for okio.AsyncTimeout)

"pool-22-thread-1" #75 prio=5 os_prio=0 tid=0x00007f40942c3000 nid=0x68bf waiting on condition [0x00007f4084900000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000e4860618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

...

If you have problems that numproc is often exceeded then you can now try to find the threads that shouldn’t be there. E.g. despite I had limited Jettys threadpool it used more and more threads. The reason was that one of my Servlets used a Timer which was only used once but the system didn’t release it after it wasn’t used any more. So more and more Timer threads were created. The solution was to use a ScheduledExecutorService instead as explained on StackOverflow.

Another limitation is often numtcpsock. On my server it is limited to 1800 connections. If that limit is exceeded one cannot connect to the server any more e.g. via ssh. By default jetty does not seem to limit the number of connections it accepts. To limit it you can add these lines to the start.ini file for Jetty 9.4:

jetty.ssl.acceptQueueSize=200
jetty.http.acceptQueueSize=200

--module=connectionlimit
jetty.connectionlimit.maxConnections=200

It limits the number of connections to 200. Without it my server accepted more and more connections without being able to process them in time. After a short time the numtcpsock limit was exceeded and the server was not usable any more. Even my existing ssh connection broke down and I could not connect any more. But with this limit of 200 connections everything seems to work fine.