July 14, 2005

Scaling, performance, and plain old bug fixing

What a couple of months this has been! First, some stats on what’s going on in the blogosphere. Technorati is now tracking over 13.3 Million blogs, and 1.3 billion links. We are seeing over 900,000 posts per day on average, which means we're adding about 10 posts per second. We’re also seeing about 80,000 new weblogs created each day. That’s more weblogs created each day than there were total when I started the service in November 2002. And our search traffic has increased by over 40% month on month for each of the last 4 months. The day of the london bombings we saw over 1.2 Million posts, and had an additional 30% increase in traffic as people turned to weblogs, moblogs, and other citizen’s media for instant updates on events in London, survivor accounts, and sharing of deep feelings on the tragedy.

Recently a number of people have had some pretty public complaints about some of Technorati's services. Thanks for the terrific feedback and comments. I feel your pain.

We sat down, listened hard to what you were saying, and then got to work. And tonight, we rolled out a raft of bug fixes and performance enhancements that should help most, if not all of the Cosmos (URL) searches you do on Technorati. It will also help with the speed of all searches across the site.

Give it a go - Here's the results for this blog, and here's how you can give it a try.

These improvements don’t fix everything - some searches are still slow, and while we pride ourselves on completeness and fast index times, there’s still a long way to go. Performance and scalability improvements are our number 1 priority until this is fixed.

For those of you who have been having problems, we're working our butts off to win back your trust. I hope that the fixes we rolled out tonight and will roll out over the coming days and weeks will be of service to you. Because, in the end, that’s what this is all about to us - to be of service to all of you.

Thanks again for the great feedback and comments. Keep us on our toes.

Posted by dsifry at July 14, 2005 1:59 AM | TrackBack | View blog reactions
Comments

Hi David, my trackback did not show up here, so here is the comment: http://hmestrum.blogs.com/my_weblog/2005/07/strange_behavio.html

"Strange behaviour of Technorati. David are you listening?

Strange! My postings do not show up anymore in Technorati. Also the postings of Maarten do not. I saw this some days ago when I posted the first Blogstorm post. Tagged with the blogstorm tag. It does not show up..."

Posted by: Hans on Experience at July 14, 2005 3:38 AM

Dave, I'll be quite honest with you... I was one of the ones blogging about this that Doc called out. But I wanted to personally say thanks for giving a great lesson on damage control and doing whatever it takes to keep your customer [users]. I put my thoughts this am.

Good job and good luck!

~jason dowdell

Posted by: Jason Dowdell at July 14, 2005 5:09 AM

Dave, I'm worried you're being out-executed by your competitors. Take a look: http://radio.weblogs.com/0001011/2005/07/14.html#a10642

Posted by: Robert Scoble at July 14, 2005 5:45 AM

Thanks David. You are really listening! Great. I hope the problem is solved soon.

Posted by: Hans on Experience at July 14, 2005 5:48 AM

Thanks for the update. It's a vast improvement for me when I do my technorati bruised-ego surfing. Now instead of a plain page with the URL/Squid error message, I consistently get a nicely designed page with the following information:

2020 Hindsight
Sorry, we couldn't complete your search because we're experiencing a high volume of requests right now. Please try again in a minute or add this search to your watchlist to track conversation.

Ah well. Keep at it. Reminds me of my psych behavior conditioning class: Behavior that is intermittently reinforced (like the Technorati search cosmos results appearing every-once-in-a-while) is the hardest behavior to extinguish. Thus I keep trying. Pathetic, that. The upside for you is that I keep trying. The downside is that my level of dissatisfaction stays elevated.

But thanks to Scoble, I'm trying out the bloglines citations thing (tho I notice that most recent results are not identical)

Posted by: Susan Kitchens at July 14, 2005 10:24 AM

Dave: That's just great! However, I do agree with Scoble... Technorati indexing needs to improve quite a bit!

Posted by: Mikhail at July 14, 2005 10:29 PM

Scoble's comparison between Technorati and Bloglines is totally misleading. He confounds sites with links, and quantity with quality. See: http://padawan.info/weblog/weapons_of_mass_delusion.html

Posted by: padawan at July 17, 2005 4:23 AM

Hi Dave!! I thought you should know that Technorati is still ignoring most of the pings submitted to it; there are sites that have linked to mine for MONTHS, that ARE being updated, and that I personally have pinged with both Technorati and blo.gs over and over, that are still not showing up on my Technorati listing. Every so often, a ping DOES result in a site being scanned and showing up on my list, but I'd peg that a a single-digit % of total pings.

The Technorati tags aren't being picked up consistently either... but that's a whole 'nother story.

Posted by: Omni at July 18, 2005 12:48 AM

Ditto, Omni. It was working for me until some point last week and now my posts are either not showing up on Technorati at all, or they have the wrong URLs and tags with each post. :-(

Posted by: Ruby at July 19, 2005 1:22 PM

Hmmm! Now I'm getting really upset. I don't have anything to complain about tonight. Technorati seems to be working very well. Congrats to you and the team!

Posted by: Randy Charles Morin at July 26, 2005 10:34 PM

Hi, David,

Thanks for the reply at my post http://blog.rex.la/archive/2005/07/31/1821.aspx

My site at Technorati.com still not update...

do a site search to technorati will reflect the situation.

http://www.technorati.com/search/blog.rex.la

it shows my site is updated 7 days ago, while I just push this post couple hours ago...

may I ask why is the cause of this problem??

thanks.

Rex

Posted by: Rex Tang at July 31, 2005 7:36 PM

Hi, David,

Thanks for the reply again at my post http://blog.rex.la/archive/2005/07/31/1821.aspx

and Yes, David,

It's working now, thanks a lot!!

Technorati is really listening, wow!!!

I am really impressed...
Now a more big fan of Technorati....


Rex.

Posted by: Rex Tang at July 31, 2005 9:15 PM