The rush after the aircraft crash and performance of server

Yesterday the crash of the Boeing Dreamliner in India caused a massive rush towards AVH laying bare a performance problem with the server. The rush is still ongoing, currently still up to 200 requests per second (!), about 17 million requests per day, and the server still operates at its limits (and beyond). It is still possible, that you can't connect to the server at first try, it is still possible, that the Apps can not fetch the content on first try.

Please do not retry to refresh the page, but wait until the relevant error message has come up and the browser/app have given up trying to download the data - your request may well have been stored in the waiting queue and might be served on first try within about 60 seconds despite intermediate information that e.g. the connection has been reset.

The performance problem, that caused pages take up to 7 minutes to be served, was the result of my work trying to stabilize the server during its frequent crashes between March 5th 2025 and May 10th, which were caused by two independent issues: a third party software module not working as expected, and a hardware problem of the hard disk (RAID) controller, which got replaced on May 10th, both changes finally fixed the crashes (the server has not been down since, now running stable for over a month).

The faulty module required to change version of it, which further necessitated to upgrade OpenSSL from 1.1 to 3.x. In order to rule out any possible vulnerability I also upgraded the webserver, which also required an upgrade of the Apache Server from 2.2 to 2.4. All of that was successful and seemingly worked well in normal operation and normal traffic load.

Yesterday however it turned out, that the server could not deliver the expected performance under stress. While normally using the full bandwidth of our leased line, it just put out about 10-20% of the capacity, driving the server further into the edge. So instead of the capacity of the leased line being the limiting factors, the server itself became that limit.

In researching and trying to optimize the server I changed the operations mode of the server, which didn't help but seemingly did not make it worse, too, average access to a page still took 6+ minutes.

Further research showed, that OpenSSL 3.x has a serious and significant performance problem due to a new architecture, which can't be overcome. So I needed to go back to OpenSSL 1.1. for the webserver while maintaining Apache 2.4 (which fortunately is possible).

On my testserver I compiled the relevant versions and tested the setup, then applied it to the AVH server.

I also needed to undo the changed operations mode of Apache, as the server still struggled other than until April 2025 (when I had upgraded all the server software) making clear the other mode still restricted the performance and made the server the limit. After returning to the previous operating mode the server returned to its normal performance, making the leased line into the Internet the limiting factor and no longer the server itself.

In short, Apache 2.4 is back to pre-fork (not the recommended worker mode) for heavy duty work with OpenSSL 1.1 to avoid the performance bottlenecks OpenSSL 3.x caused.

Once again my plea: please do not press refresh before the browser tells you that the connection has failed. Once the error message has occurred (on browser and/or App), feel free to retry, the second or third try - depending on current server load - might now succeed again.

I am deeply sorry for these problems and the trouble you had and still might have accessing our server.

Update Jun 14th 2025: In the meantime it became clear, that the server performance was still extremely poor after a while, while performance immediately after a restart was okay and completely stable for a while. Even without hitting any extremes (memory, requests), with the load entirely stable, the server with Apache 2.4 suddenly became extremely slow for no apparent reason. As result, I have now re-instated the previous server Apache 2.2, that has been proven over the years and many more rushes and which seems to be working well again (fingers crossed).