Technorati Growing Pains

These past weeks have been a pretty busy one for me and the growing Technorati team. Before I get too far in this post, I've got a mea culpa - Technorati hasn't been very responsive lately. Fact is, we've been getting a lot of attention and new searches, and the blogosphere seems to be growing at a pretty steep rate as well. This double whammy has caused our current infrastructure to buckle, and has caused some service outages.I'm sorry. Here's what we're doing to fix it: I've got a new, much more scalable infrastructure designed and currently being built. I'm committed to having it up and running by the end of the month, just in time for Technorati's first anniversary. This will be the third generation of our infrastructure, each designed to be more scalable and flexible than the last. After stability, the next priority is response time - we're gunning for a response time of under 1 second. Allow me to give you some growth statistics: One year ago, when I started Technorati on a single server in my basement, we were adding between 2,000-3,000 new weblogs each day, not counting the people who were updating sites we were already tracking. In March of this year, when we switched over to a 5 server cluster, we were keeping up with about 4,000-5,000 new weblogs each day. Right now, we're adding 8,000-9,000 new weblogs every day, not counting the 1.2 Million weblogs we already are tracking. That means that on average, a brand new weblog is created every 11 seconds. We're also seeing about 100,000 weblogs update every day as well, which means that on average, a weblog is updated every 0.86 seconds. So, for those of you who have written to me wondering about the recent outages, again, I'm sorry. Keep the feedback coming, btw - we really appreciate it. And if you have written lately and no one has responded, we haven't stopped caring - we've just been really busy fighting fires and getting the new infrastructure built. Don't hesitate to drop me a line if you want to express something privately.