Technorati Growing Pains
12These past weeks have been a pretty busy one for me and the growing Technorati team. Before I get too far in this post, I’ve got a mea culpa – Technorati hasn’t been very responsive lately. Fact is, we’ve been getting a lot of attention and new searches, and the blogosphere seems to be growing at a pretty steep rate as well. This double whammy has caused our current infrastructure to buckle, and has caused some service outages.
I’m sorry.
Here’s what we’re doing to fix it: I’ve got a new, much more scalable infrastructure designed and currently being built. I’m committed to having it up and running by the end of the month, just in time for Technorati’s first anniversary. This will be the third generation of our infrastructure, each designed to be more scalable and flexible than the last. After stability, the next priority is response time – we’re gunning for a response time of under 1 second.
Allow me to give you some growth statistics: One year ago, when I started Technorati on a single server in my basement, we were adding between 2,000-3,000 new weblogs each day, not counting the people who were updating sites we were already tracking. In March of this year, when we switched over to a 5 server cluster, we were keeping up with about 4,000-5,000 new weblogs each day. Right now, we’re adding 8,000-9,000 new weblogs every day, not counting the 1.2 Million weblogs we already are tracking. That means that on average, a brand new weblog is created every 11 seconds. We’re also seeing about 100,000 weblogs update every day as well, which means that on average, a weblog is updated every 0.86 seconds.
So, for those of you who have written to me wondering about the recent outages, again, I’m sorry. Keep the feedback coming, btw – we really appreciate it. And if you have written lately and no one has responded, we haven’t stopped caring – we’ve just been really busy fighting fires and getting the new infrastructure built. Don’t hesitate to drop me a line if you want to express something privately.
Related posts:
I totally agree with Seb, David. Thanks for giving us an extremely useful tool!
Technorati is the one addition to my serverlogs that help me keep track of who’s linking to me, and thus helps me find new people to enrich my social networks, and bring me new opinions and paths of thought.
Thanks!
It might lessen the email stack if a link to this post was featured on the main page of technorati.
This is a wonderful service, David. I have some thoughts on ‘add-ons’ that might (while further taxing your infrastructure) allow you to generate a bit more revenue for your trouble — drop me a line. Also, I second what vanderleun said — a simple note on the affected pages explaining why services are down would help a lot.
Since you are the definitive source for these number and people start misquoting the number of tracked weblogs as the number of active weblogs, can you count all weblogs that are older than one month and that have been updated during the last two or three weeks? This should weed out the test sites that people create on various platforms before the settle with their permanent weblog.
You guys need more money. Serious growth is just starting…triple your expectations, get a CEO you like and is well connected and grow. Technorati is the web part, of the WWW. Think about how your service will be used inside corporations to understand their social networks.
Best.
I just want to echo what everybody else is saying, David – you’re doing a great job!
Growing is definitely a pain. I am right along with ya man. Keep it up man. Keep it up. But when you get some time send Dan down here so we can finally get that beer
we can be patient! Keep up the great work
David, Your dedication and fantastic service is legendary in the blog world. I read a rumour yesterday that some VCs with deep pockets were eyeballing technorati. If that’s what you’re looking for, I wish you the best! You deserve a big payoff for all of your hard work.
David, well pleased with the service, and hoping your growing pains ease.
Not sure what your business model is, but I don’t mind admitting that the Technorati report is my only “must visit” page every time I check out my own blog. I learn a lot about what people are hitting on and actually find leads to all sorts of sources and search patterns I wouldn’t have found or tried myself. Without trackback, this is an invaluable service for me.
Good luck and keep it up.
Technorati is a total inspiration! Thanks for keeping us all informed about what is happening, and keep up the good work!
take your time dudes, it’s not the end of the world! we very much appreciate what you do for us.