Monthly Archives: March 2011

Last system update aftermath

In the beginning of this month we performed a non-trivial update to the system in order to increase performance and give you some extra candy. With more PLR articles, of course.

However, in the first few days of running the new version, things weren’t looking bright. Our application servers were hanging every few hours for no apparent reason. We temporarily mitigated this problem by running several application server in parallel and restarting them every hour before they would hang.

After days of investigating what went wrong we finally nailed it: a teeny tiny bug that under “perfect storm” conditions caused the server to go into an infinite loop. We do extensive testing before publishing a new version of the system to prevent problems like this. However, we only test the system with logged-in users, not with anonymous users -> this enabled the “perfect storm” conditions to sneak up below our radar: when anonymous users opened the /articles/search/ URL, our application tried to do some internal checking for which JavaScript, got permission errors and hanged inside a loop. Scary. But easily fixable.

Anyhow, the patched version has now been online for a few days and the system is back to it’s stable state, performing better than ever. Sorry for any inconveniences and enjoy using Big Content Search!

If you have any questions let us know.