Designing systems to scale

Historically when an application gets more popular, we as developers simply make the server a bit bigger. But what happens when you can’t make the server any bigger? Suddenly you have an app that’s getting millions of hits a day and a database that’s a terabyte big. Imagine how hard it’d be to find a way for more people to sign up whilst not impacting the site’s performance for your existing customers. It’s not easy (we’ve done it!). Wouldn’t it be easier to plan for scale from day zero?

Thanks to services such as Amazon AWS, it’s typically cheaper to run several smaller servers than one big one, so as well as easy scaling you’ll be saving money. Great applications need great foundations. Michael’s taken Tweetdig.com from struggling to process a few thousand tweets to processing over a million per day, and has learned a lot along the way. He’d like to share what he’s learned so far and the benefits and pitfalls of the various approaches that he’s tried.

Slides

Video