Our illustrious VP Of Engineering, Adam Hertz, has just posted a progress report on the service improvements , scalability increases, and performance gains that have been going on under the covers over the last couple of months, extensions to the work we began back in July and reported upon at the beginning of September.
More to come. Looking forward to seeing folks at the Web 2.0 conference next week - Technorati, along with del.icio.us, Flickr, Flock, Odeo, wink, and WordPress are all hosting a party on Thursday October 6 from 9PM at Swig at 561 Geary Street in San Francisco. Please stop by if you're in the city!
Technorati and Edelman are partnering in an attempt to better understand how blogging and traditional PR intersect, and what bloggers think about communication from mainstream companies. Edelman is a global public relations firm representing brands such as Xbox, Nissan, and Dove. Working together, we created a 18-question survey to better understand the blogging community and your preferred methods of hearing from companies.
Is receiving a press release from a PR agency just more spam? What about product discounts or free goods? Are there better ways for traditional marketers and bloggers to interact? What is the implicit contract created when marketers and bloggers communicate? What are the ethical questions? What are companies not listening to that they should be listening to?
Please take a few minutes to answer the survey. This survey is intended as a starting point for discussion, and not a comprehensive be-all, end-all survey. Personally identifiable information is not tracked but you may explicitly give us permission upon submission to send you the final results of the survey. We will review the aggregated results and include the findings in a public white paper next month to help inform bloggers, companies, and public relations firms. All survey respondents have the option to receive this white paper via e-mail as soon as it is available.
The blogosphere is abuzz with Google's launch of their Blog Search. So far things look pretty interesting, and having a big traditional search player like Google working on blog search is a validation moment for the entire blogosphere.
This will mark a major milestone for the World Live Web. At Technorati, we have a tremendous amount of respect for the Google team and for everything they've done in the world of search. I'm sure that they'll continue to improve over the coming months, perhaps including tags, recent images and links, zeitgeists, blogger tools, and other types of semistructured data. I'm sure that they'll also start indexing the full-text of blog posts, not just the partial text found in most blog feeds.
I welcome the competition. We've got some tricks up our sleeves too - and there's no doubt that in the end, the competition will end up producing more innovation and better services for bloggers and readers.
Welcome to the party, Google!
So, while all of our infrastructure work progresses and our backend search and infrastructure engineers are busy upgrading and scaling the service, Technorati's front-end designers and developers cooked up a fun new feature: Technorati Blog Finder.
Blog Finder helps answer the question, "How can you find authoritative blogs on a subject?"
Derek Powazek posts about Blog Finder on the Technorati Weblog. My favorite feature is that it allows you, the blogger, to tag your blog, and add yourself to the categories that you want to be listed under. We've made some educated guesses based on how people tagged their posts over the last 6 months - and over 2 million blogs are already included in the index, but now you have the power to add yourself to the directory in a few simple steps.
It's been a long and busy month, and I wanted to give y'all an update on the infrastructure, performance and scalability progress over at Technorati. There's been a lot going on as I described earlier in the year, and we've made some progress, but there's important things that are still broken, and are being fixed this month.
The situation as of couple of months ago
The blogosphere has been growing at an explosive rate - Technorati is now indexing over 16 million blogs, with about 100,000 new blogs created every day. And there's over 1.4 Million new posts every day, and about 22% of those posts are from spam or fake blogs, which means that even after we pull out the spam and fake blogs from the indexes, we are dealing with about 1.2 Million posts each day.
We just weren't expecting that kind of sudden growth, both on the posting side and also on the search side, and frankly we didn't plan well enough to handle the load. We've been adding new machines to our datacenter, - over 400 now - and more coming each week, and we've been fixing bugs and making performance enhancements on the web site as well.
However, Cosmos search (or URL search) is still being worked on, and is often timing out under the increased load. Unfortunately this is also one of the searches that bloggers find most compelling, as it helps you to all know who is linking to your blog, and it is the very first type of search that Technorati made available, so it is near and dear to our hearts. Everyone here also uses it every day, so it really sucks when it isn't working right.
As search traffic has grown, we've also seen an increase in support and feedback requests. It's my goal to make sure that we respond to all support requests within 24 hours of getting the request. right now, we're not meeting those goals, and some people haven't had a human response in over a week from when they sent in their request.
What we're doing
Once we got our keyword search infrastructure back on track, our infrastructure team has been working 100% on fixing Cosmos search. Our current plan is to have Cosmos search back up and running by the end of September, and you'll see incremental improvement throughout the coming month. I'll keep you informed on progress of this critical project. As the project progresses throughout the month, you'll be able to see progress because you'll see fewer and fewer error messages when you do a URL search as September progresses.
We're busy expanding out our support capabilities, and also putting together tools to make it easier for users to help answer their own questions before a Technorati support staffer has to get involved, and we've already made a bunch of fixes and feature enhancements to help fix the most common support requests, like fixes in our blog claiming code.
What about new stuff?
While we work on these core infrastructure issues, we're not resting on our laurels in our dedication to provide great tools and services for bloggers and for people who want to keep track of what's happening on the web right now. There'll be more to announce in the coming days and weeks, stay tuned...
Thanks for your support
I am consistently humbled and amazed at how great our users are. You guys have stood by us as the service has grown and has gone through growing pains. We take this trust very seriously, and are working very very hard to live up to your expectations. Thanks.