As you know, Neowin upgraded its backend Content Management System (CMS) on Friday January 15.
We also upgraded the forums to Invision Power Board 3. Unfortunately we had to roll back the changes a few days later due to extremely high loads on our server, specifically on our forum install. Our CMS (Ignition) remained online throughout the load issues and performed extremely well.
The same can't be said about our forum install though; each time we tried to place it back online it would spike with load forcing us to pull it offline again. One would have assumed this was the fault of Invision Power Board but it wasn't. We determined that the problem was spiders from Google, Bing and Yahoo were hitting us at around 10 times every second. When we upgraded to IPB3, due to our completely new URL structure, the spiders freaked out at all the new links. The spiders went into overdrive trying to re-index not only the 9 million posts we have, but also the news. Fortunately we do not have 9 million news posts so the news pages suffered less.
The idea of spiders causing the problem had been suggested before by a member of the Invision Power Services support team, but we never fully investigated it. After all, our spider traffic has always been really high. However, yesterday, in an attempt to do a load test, we put IPB3 onto our dev server and included it in an iframe on every page of the live forums. In non dev speak, every time someone loaded a page on the live site for a brief period yesterday, the development forums would get hit. Theoretically, that would recreate the load we saw over the weekend. However... our load stuck at around 0.3. This result puzzled us for a little while until we thought about it a little bit: spiders do not index iframes. While we had the same number of members on the live forums as the dev forums, the spiders weren't hitting it at all.
We knew we needed a solution, and we found one: tell the spiders to index less frequently. We amended what's called a "robots.txt" file to tell all search spiders to only index us a maximum of once a second. Our hits lowered, and we decided this might be the solution we were looking for: load was falling on IPB2, we couldn't imagine the effect it may have on IPB3.
This led to yesterday's successful load test. For 20 minutes, we put up a live version of IPB3, using the database we generated over the weekend. After Simon fixed a misconfiguration we had (which was causing the forums to stall and the load to spike initially), our loads fell. The forum was speedy. Essentially, we determined that it was fixed.
We have now successfully moved to Invision Power Board 3 and would like to thank you all for being extremely patient during last weekend's downtime. Neowin would also like to extend its thanks to Brandon from Invision Power Services who helped us determine the exact issue.
So this time, welcome to Ignition and Atlas :)
Neowin Developer Simon Andrews contributed to this report
144 Comments - Add comment