Scaling, performance, and plain old bug fixing

Scaling, performance, and plain old bug fixing

Scaling, performance, and plain old bug fixing

What a couple of months this has been! First, some stats on what’s going on in the blogosphere. Technorati is now tracking over 13.3 Million blogs, and 1.3 billion links. We are seeing over 900,000 posts per day on average, which means we’re adding about 10 posts per second. We’re also seeing about 80,000 new weblogs created each day. That’s more weblogs created each day than there were total when I started the service in November 2002. And our search traffic has increased by over 40% month on month for each of the last 4 months. The day of the london bombings we saw over 1.2 Million posts, and had an additional 30% increase in traffic as people turned to weblogs, moblogs, and other citizen’s media for instant updates on events in London, survivor accounts, and sharing of deep feelings on the tragedy.

Recently a number of people have had some pretty public complaints about some of Technorati’s services. Thanks for the terrific feedback and comments. I feel your pain.

We sat down, listened hard to what you were saying, and then got to work. And tonight, we rolled out a raft of bug fixes and performance enhancements that should help most, if not all of the Cosmos (URL) searches you do on Technorati. It will also help with the speed of all searches across the site.

Give it a go – Here’s the results for this blog, and here’s how you can give it a try.

These improvements don’t fix everything – some searches are still slow, and while we pride ourselves on completeness and fast index times, there’s still a long way to go. Performance and scalability improvements are our number 1 priority until this is fixed.

For those of you who have been having problems, we’re working our butts off to win back your trust. I hope that the fixes we rolled out tonight and will roll out over the coming days and weeks will be of service to you. Because, in the end, that’s what this is all about to us – to be of service to all of you.

Thanks again for the great feedback and comments. Keep us on our toes.

Sifry's Alerts
About The Author
I'm founder of a number of companies, including Offbeat Guides and Technorati. I was the cofounder and CTO of Sputnik and Linuxcare, founding board member of Linux International, and a WEF Technology Pioneer. I've been around the block a few times. Some might call me a serial entrepreneur. You can contact me at


  • By Hans on Experience 14 Jul 2005

    Hi David, my trackback did not show up here, so here is the comment:
    “Strange behaviour of Technorati. David are you listening?
    Strange! My postings do not show up anymore in Technorati. Also the postings of Maarten do not. I saw this some days ago when I posted the first Blogstorm post. Tagged with the blogstorm tag. It does not show up…”

  • By Jason Dowdell 14 Jul 2005

    Dave, I’ll be quite honest with you… I was one of the ones blogging about this that Doc called out. But I wanted to personally say thanks for giving a great lesson on damage control and doing whatever it takes to keep your customer [users]. I put my thoughts this am.
    Good job and good luck!
    ~jason dowdell

  • By The Newest Industry 14 Jul 2005

    Technorati: Performance Fixes?

    Dave Sifry from Technorati very publicly announce a mea culpa, and said that he and his team rolled out some performance enhancements that should resolve the problems.[here]
    Dave, they’re still there. Go to GrabPERF and select any of the Technorati …

  • By Robert Scoble 14 Jul 2005

    Dave, I’m worried you’re being out-executed by your competitors. Take a look:

  • By Hans on Experience 14 Jul 2005

    Thanks David. You are really listening! Great. I hope the problem is solved soon.

  • By Susan Kitchens 14 Jul 2005

    Thanks for the update. It’s a vast improvement for me when I do my technorati bruised-ego surfing. Now instead of a plain page with the URL/Squid error message, I consistently get a nicely designed page with the following information:
    2020 Hindsight
    Sorry, we couldn’t complete your search because we’re experiencing a high volume of requests right now. Please try again in a minute or add this search to your watchlist to track conversation.
    Ah well. Keep at it. Reminds me of my psych behavior conditioning class: Behavior that is intermittently reinforced (like the Technorati search cosmos results appearing every-once-in-a-while) is the hardest behavior to extinguish. Thus I keep trying. Pathetic, that. The upside for you is that I keep trying. The downside is that my level of dissatisfaction stays elevated.
    But thanks to Scoble, I’m trying out the bloglines citations thing (tho I notice that most recent results are not identical)

  • By Mikhail 14 Jul 2005

    Dave: That’s just great! However, I do agree with Scoble… Technorati indexing needs to improve quite a bit!

  • By The Basement 14 Jul 2005

    Size (of the whole Blogosphere) Doesn’t Matter

    Yesterday in Stop
    the Blog Statistic Madness! I touched on something I’ve been
    meaning to blog about for a while now – your base. I’m not
    talking about the base of the classic game Zero Wing, I’m talking
    about the base in the market research sense. …

  • By TechCrunch 16 Jul 2005

    Web 2.0 This Week (July 10 – 16)

    The main Web 2.0 news this week focused on Technorati (is it dyin or is it rockin?) and the release of Atom 1.0. Lots of other random and interesting stuff as well, including important chia-pet news.
    Weekly Summary:
    1. Technorati is Dead! Long Liv…

  • By 17 Jul 2005

    Interrogations sur l’avenir de Technorati

    Technorati occupe une place

  • By padawan 17 Jul 2005

    Scoble’s comparison between Technorati and Bloglines is totally misleading. He confounds sites with links, and quantity with quality. See:

  • By lotusmedia 2.0 19 Jul 2005


    Why doesn’t Technorati ever seem to work quite right (or at least work as they claim it does)? In addition to being slow as molasses, it has recently started to incorrectly index my posts. Maybe I shouldn’t complain since a month ago it …

  • By Ruby 19 Jul 2005

    Ditto, Omni. It was working for me until some point last week and now my posts are either not showing up on Technorati at all, or they have the wrong URLs and tags with each post. :-(

  • By 19 Jul 2005

    Interrogations sur l’avenir de Technorati

    Technorati occupe une place

  • By Alex Barnett blog 19 Jul 2005

    RSS will succeed by disappearing

    Blogspotting quotes Dave Sifry, CEO of Technorati:
    “Sifry: I think RSS needs to be completely transparent….

  • By 20 Jul 2005

    Interrogations sur l’avenir de Technorati

    Technorati occupe une place

  • By Randy Charles Morin 26 Jul 2005

    Hmmm! Now I’m getting really upset. I don’t have anything to complain about tonight. Technorati seems to be working very well. Congrats to you and the team!

  • By Rexiology... 31 Jul 2005

    [Trackback] Technorati is not linking or tagging my posts…

  • By Rex Tang 31 Jul 2005

    Hi, David,
    Thanks for the reply at my post
    My site at still not update…
    do a site search to technorati will reflect the situation.
    it shows my site is updated 7 days ago, while I just push this post couple hours ago…
    may I ask why is the cause of this problem??

  • By Rex Tang 31 Jul 2005

    Hi, David,
    Thanks for the reply again at my post
    and Yes, David,
    It’s working now, thanks a lot!!
    Technorati is really listening, wow!!!
    I am really impressed…
    Now a more big fan of Technorati….

  • By Rexiology... 31 Jul 2005

    [Trackback] Technorati is not linking or tagging my posts…


Follow @dsifry