AppStream, the little app with massive impact – autopsy of yesterday’s server crash

A while back, while chasing our tail for months waiting for our Appsfire 2.0 app to be approved (or rejected) by Apple [note: it is now approved http://getap.ps/appsfire], we decided that we should start producing subset apps, just to see if Apple would approve them. This was really mostly a proxy effort. Imagine that! We wrote apps just to get a reaction from Apple. Not exactly your classical case of app engineering!

Nevertheless, this is how Appstream for iPad was born (http://getap.ps/appstream), mostly reusing parts of the code from Appsfire 2.0 – except that this app would do mostly one thing: display a spiralling stream of apps that are currently being discovered and clicked on various Appsfire properties (our site, our widgets, our search widgets, our yet-to-be-approved appsfire app, etc…).

Now, this app was built hastily, along with its counterpart, the corresponding web-service (i.e. the server-side logic that sends fresh stuff to the app).
This web-service boiled down to a simple and innocuous database request that extracts roughly 500 apps recently clicked on our properties, removing duplicates to bring additional diversity.
Now, most database requests are optimized and perform in fractions of a second. This one request took about several seconds… possibly 50x too long.
All we wanted was for Cupertino to give it a try… no big deal right, a few seconds for a screen to load is okay right? at worse, we’d get a reassuring “sorry your app is rejected because it’s too slow”.

Zoom past a few weeks. Our traffic goes up, Appsfire 2.0 eventually gets approved, we add complexity to our site and new great partners that naturally test our ability to scale. So far so good.
And then comes Appstream, that little big app with lots of spinning icons, it’s cool, it’s mesmerizing. People seem to like it. It quickly gets featured in the top #5 free iPad apps in France and top 40 in the US, as predicted by http://appsfire.com/apptrends …
… and just like that, the service crashed.

Building_collapse_appstream

That database request came to bite us back. On July 17th, this little app hit us many tens of thousands of times with a complex and slow and non-optimized server-side request, bringing our database server to a grind. It took us a while to realize that Appstream for iPad was the culprit and an unexpected runaway attraction!
The query is now optimized, nice and speedy. All services are check.

Anyways, just a round about way to say: dear users and partners, sorry we crashed – and everytime we hit a wall, we learn, and we try harder. Please accept our sincere apologies, and thank you for your patience today.

Appstream for iPad: http://getap.ps/appstream.
Give it a spin if you haven’t already.
This entry was posted in Uncategorized. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.