September 30, 2004

Technorati Hackathon, San Francisco: Wed Oct 6, 2004

We're having a Hackathon at our new offices in San Francisco, on Wednesday, October 6. We're going to have lots of pizza and beer, and lots of outlets and free wifi. The idea is to actually do some real web services hacking that night after talking about it all day at the looks-to-be-great Web 2.0 conference. If you're a hacker who knows about our API, or is interested in learning more about coding web services, this is going to be a chance to hang out with our core developers as well as with other leading web services developers, with the goal being to foster great new applications and tools using those APIs. We're also really interested in sparking further conversation and getting feedback from you so that we can make Technorati more valuable for you. You don't have to be a Web 2.0 attendee to come to the hackathon - but there's sure to be lots of attendees there.

OK, the details:

WHEN: Wednesday October 6, 2004, from 8PM - whenever!
WHERE: Technorati Offices (map) at 665 3rd Street, Suite 207, San Francisco, CA 94107 (between Brannan and Townsend Streets)
WE PROVIDE: Free pizza, beer, soft drinks; Fast WiFi, whiteboards, lots of room to hack as a group or individually, experts, help, and advice.
YOU PROVIDE: Creativity, energy, good humor, great ideas, willingness to teach and learn, readiness to hack!

IMPORTANT NOTE: Space is limited (our offices only hold so many people!), so please RSVP ( as soon as possible to guarantee your spot! First come, first served.

Posted by dsifry at 7:07 PM | View blog reactions

New Technorati Toolbar for Firefox

Thanks to the cool guys at UltraBar, you can now have Technorati at your fingertips with the Technorati Toolbar! I've been using this for a while now, and it is fantastic - you can type in search queries, and get back up-to-the-minute results from around the web, and you can also click on the Technorati talk-bubble () from any page you're browsing to see who is talking about that page, and what they're saying - which has made reading the news online a completely more satisfing experience. The toolbar works on all versions of Firefox past 0.9 (including the 1.0 PR that is currently available). Fantastic job, guys!

Posted by dsifry at 6:20 PM | View blog reactions

September 27, 2004

And we're back

The Technorati service has now been restored following the weekend from hell. Many thanks to the Engineering and Ops teams who worked all weekend to get things back up and running quickly. Thanks as well for the kind response from folks out in the blogosphere, and from our customers and partners. We're 100% committed to providing great service to all of you.

Onward and upward.

Posted by dsifry at 1:38 PM | View blog reactions

September 26, 2004

Putting out colo fires (literally!), aka “A weekend of downtime”, grrr.

Ugh. What a horrible weekend. I and the team have been spending the entire weekend dealing with massive data corruption caused - of all things - by an electrical fire on the main electrical line coming into our colo here in San Francisco.

The colo fire has led to a cascade of failures that has caused the Technorati service to be down for most of the weekend. It's also giving me a lot more respect for people who build and maintain 100% uptime of services, the trials and tribulations they go through, and also the cost of being operationally excellent.

What Happened

At about 9:30PM PST Friday, there was an electrical fire on the power main inside our colocation center, where our entire server infrastructure is housed. This caused our battery backup power supplies to kick in, but the independent power generator at the colo never kicked in - possibly because the problem was a fire inside the building rather than a general power blackout of the neighborhood. Well, the fire was only problem #1. We weren't expecting or planning for an outage of that kind. It caused a cascade of other problems that made the rest of the weekend a huge PITA. Problem #2 was that we didn't have a good enough emergency plan in place that would shut our systems down cleanly when power ran out like that. Unfortunately, that meant that when the batteries died, our server farm went down quite ungracefully - causing problem #3, which was data corruption due to the unclean shutdown.

The rest of the weekend has been spent recovering from these failures - we've had to do consistency checks and then rebuilds of the data sets that got corrupted, and we're doing that for over a hundred machines. Bad bad bad. At least we've been performing regular daily backups, and we're able to use that as starting points on our road to recovery. The current ETA to get services back up and running is by Monday morning, which will mean a weekend of unplanned downtime.

What we're doing about it

Clearly, this is unacceptable, but the damage is already done, so the big question on my mind is how to learn from this outage and make sure that it never happens again. One of the important things learned is that there's a reason why some colocation centers are called “Tier 1” (and priced that way) and others are not. Tier 1 means that everything is overprovisioned, and there's plenty of infrastructure backup already built into the place - electrical, network, fire suppression, environmental, security, etc. We have been planning a move to a new colocation center, but this most recent incident just underscores the need to move asap. Second, it illustrates a hole we had in our emergency plan - we had built our emergency plan based on a threat level of a short outage followed by a quick electrical recovery. We had planned for a shutdown of critical systems if battery power fell below a certain threshold (upsd for you techies out there), but we hadn't gotten it implemented given that we were planning the move to the new colo. That of course, led to the data corruption that is keeping the team up all weekend.

Once we get past this crisis and get the service back up and running again, I guarantee that we'll be doing a post-mortem analysis to see where the failure points were, and how we can avoid them in the future. I have learned a lot from this experience about the value (and implicit cost) of planning and building systems around unreliable components, and doing everything you can to eliminate risk - and also about planning for quick recovery when the unthinkable happens. I have a lot of respect for the folks at Google, Yahoo, eBay, and the like for their ability to build and maintain a solid world-class infrastructure, have it scale, and also innovate with new applications as well.

Planning for Murphy's Law

Count me as one of the humbled. To our users and customers: Ouch this hurts, and we're working on making sure that outages like this never happen again. To the folks down in ops and engineering at companies around the globe keeping these systems running and useful, no matter what Murphy throws at you, you've got my appreciation.

I'll post as we have more updates on service status as the day progresses.

UPDATE: As of 10:00 PST Monday, the infrastructure is back online, and we're live again, but we're still making sure that everything is back 100%. Searches should work, and we're monitoring our response time.

Posted by dsifry at 2:19 PM | View blog reactions

September 22, 2004

Sputnik releases Control Center 3.0, and hosted service: SputnikNet

The folks at WiFi management software and services company Sputnik have just released a major software and services upgrade. Sputnik Control Center is the easy-to-use, easy-to-buy software that allows you to manage hundreds of WiFi access points as a single system, manage access control, create and deploy captive portals, track usage by AP and user, set up network policies, and much much more. Check it out: there's a Sputnik Hotspot Kit for only $599. that includes Two Sputnik AP 160s and two Sputnik Control Center licenses. Makes it really easy to become a WiFi access provider, or to install secure wireless across a company or campus.

SputnikNet enables you to run a managed wireless network without having to set up or run your own server. With SputnikNet, you get a hosted Sputnik Control Center set up just for you. Just plug Sputnik-Powered APs into broadband Internet, and manage your wireless network. You can manage as many access points and wireless networks as you like for only $19.95 per access point per month.

Congratulations, Sputnik folks! Full disclosure: I'm a founder and advisor for Sputnik, so don't just take my word for it - go and see what others are saying. Daily Wireless has a good review, as does WiFi Networking News and WiFi Planet.

Posted by dsifry at 9:43 PM | View blog reactions

So what's the deal with GTD and Tinderbox?

Of late I've been hearing more and more about “Getting Things Done” (aka GTD) and programs like Tinderbox? Seems that there's some new management trend starting to spread. Anyone actually use these systems? Are they useful for long-term productivity improvement? Before I go and investigate this too much further, I'm interested in hearing if it's just another management/productivity fad, or if there's something real behind all this. Both the book and the software seem to be getting a lot of conversation spreading about them.

Posted by dsifry at 9:25 PM | View blog reactions

September 20, 2004

Creating a timeline of Blogging's influence on mainstream media

I was recently asked by a reporter for resources on real, concrete events where blogging had a significant effect on political events, and while a number of events came to the top of my head (Trent Lott, Salam Pax, Ed Schrock, Bush's National Guard memos fisking) I was thinking it would be great to have a timeline for all of us - so I started one. It is pretty bare-bones for now, with entries for the aforementioned Lott, Schrock, and Rathergate. Please drop by and add/edit the page to fill out the timeline.

Posted by dsifry at 2:16 AM | View blog reactions

September 8, 2004

Back from the RNC

Back from the RNC, and I sure am glad to be back on the west coast. It was great to see some old friends (and make some new ones) while out in NYC, getting the new Politics site a workout was exciting, and working with CNN was a joy - those folks are real pros, but it is great to be back at home.

There were lots of bloggers over in the blogger's alley that the RNC provided in the Felt Forum outside of the main convention hall. Here's a picture of some of the bloggers at the alley interviewing Ari Fleisher, former Press Secretary for the current administration. Click on the picture to see a larger version.


Posted by dsifry at 9:52 PM | View blog reactions

September 3, 2004

RNC Day 4 Wrap-up

While bloggers were a novelty at the DNC in Boston and were less of a story in and of themselves during the RNC, the quality of commentary and the number of breaking stories during the RNC show that bloggers are starting to hit their stride. The toppling of Ed Schrock, two-term Republican congressman from Virginia, after investigative reporting from a blogger showed the growing political power of citizen journalism. The use of blogging technology during protests and also from delegate phonecams on the convention floor show that this pushbutton publishing technology is filling an important journalistic niche.

The convention wrapped up as first President Bush spoke and then Senator Kerry, in an unusual move, followed from a campaign rally in Ohio. Bloggers at the convention formed into two primary camps: mostly conservative RNC credentialed bloggers, writing from Blogger's Alley in the Felt Forum, outside the main convention hall, and a group of mostly liberal bloggers that gathered at The Tank, a performance space in midtown Manhattan. Of course, there were many other bloggers from all over the country who were adding to the conversation. Top stories from the final day of the convention include:

  • Democrats divided?That's what Dick Morris wrote, in the New York Post's opinion column. " In an incredibly striking contrast, Bush voters are united on virtually all the questions that divide the Kerry vote. So Bush can advance his agenda with impunity while taking aim at Kerry voters who are antagonized by their candidate whenever he has to choose a position." Not all liberals agree. Lambert from Corrente wrote, "If Kerry can get people to listen and think, Bush is toast."

  • Zell Miller challenges Chris Matthews to a duel Chris Matthews had a heated exchange with Zell Miller after his speech Wednesday night. Here's the clip in Windows Media Format. Media Bistro has more details, and ongoing commentary on the Miller speech is here.

  • Word frequency analysis The New York Times reports via an interesting graphic the frequencies of different words used during the DNC and the RNC. Micah L. Sifry performs the same analysis on Bush's acceptance speech.

  • President Bush's speech Similar to the DNC, Conservatives went quiet after the final speech of the convention, but the Technorati Attention Index™ showed a burst in anti-Bush postings after the speech ended. this is probably because of the "satisfaction effect" - that conservatives are happy with the climax of the convention, while liberals are so unsatisfied, they are vociferously blogging. William Saletan commented after the speech, "The 2004 election is becoming a referendum on your right to hold the president accountable." Dave Winer wrote, "I'm glad Kerry responded to the Republicans. Maybe it's time, though, to consider a new format, where they do a Democratic talk show, with Mario Cuomo, Wesley Clark and James Carville reviewing the Republicans. They require serious rebuttal. Don't pretend they're going to roll over and let the Dems win. Fight fire with fire. It's good that Kerry has shown his anger. It's not good that he trailed off into his standard incoherent stump speech."

  • George Bush's National Guard Service questions Two important updates in the George Bush Alabama National Guard story - Salon reports on the widow of a Bush family confidant who says that Bush did no National Guard service in the spring of 1972. And Ben Barnes, the former Texas official who says he pulled strings to get George W. Bush into the Air National Guard will be appearing on 60 Minutes.

Posted by dsifry at 1:19 AM | View blog reactions

September 2, 2004

RNC Day 3 Wrap-up

The heat turned up in Manhattan today, as speeches by Zell Miller and Dick Cheney provided red meat to party faithful, and protests outside the convention hall increased in intensity and number. After-hour parties abounded, but behind the scenes, the effect of money in politics only got rarely reported. More from on and off the floor:

  • Reactions to Zell Miller: The angriest speech of the convention, seemingly came from Zell Miller, as Glenn Reynolds notes, "It's funny that the purest voice of Jacksonian America at this Republican convention -- in fact, at either convention -- comes from a Democrat.". Begging to Differ wrote, "Zell Miller was more effective tonight than any Republican could have been. John Kerry will have to answer, if he can."

  • The two Cheneys - diverse reaction from liberals and conservatives The Technorati Politics Attention Index™ saw heavy blogging from both the left and right regarding Vice President Dick Cheney's speech tonight. Conservatives like Sister Toldjah remarked, "A SOLID performance!". Ann Althouse wrote, ""He lays it out. And you can take it or leave it. He's not doing the twist. He's Dick Cheney." Liberals felt differently. Josh Marshall described a common liberal perspective: "My first thought was, bold words for a man whose office is the subject of an on-going criminal inquiry. But apparently that’s not the subject of polite conversation."
  • Arnold gets fact-checked The liberal blogosphere has been buzzing about Arnold Schwartenegger's speech last night. Numerous bloggers pointed out that while Arnold has spoken of Nixon debating Humphrey in 1968, the debate never happened.

  • Protest Vignettes Outside of the convention, New York is filled with protests, from big, to small, to personal. Inside the convention hall, AIDS demonstrators disrupted a Republican youth gathering on the floor, unveiling an anti-Bush sign and disrupting a speech by Andrew Card. Andrew Sullivan reports on a particularly rude protester, while Peter Northrup wrote "of a self-proclaimed liberal [that] was spending her day, not protesting with angry slogans, but sitting in front of an empty chair and a sign that invited conservatives to sit down and talk with her about the future of the country." Ratherbiased has pictures of a protester being removed from tonight's Cheney speech.
  • Blogging a talk-show appearance: Michelle Malkin blogs her side of the story after appearing on MSNBC's Hardball. First-hand accounts like this are shifting the fulcrum away from show hosts and producers who often attempt to unfairly create controversy. Malkin's account, while completely subjective, allows her to get her side of the story out to the world, just as the FCC's Michael Powell or Mark Cuban, the owner of the Dallas Mavericks, use their blogs to speak to the world in a timely, unedited fashion.

Posted by dsifry at 12:06 AM | View blog reactions

September 1, 2004

Wired News goes behind the scenes of the conventions - what we learned

Adam L. Penenberg at Wired News just published a great article covering the lessons learned by bloggers (and me personally) at the DNC, and the changes we made to the Election Watch 2004 site, including the reorganization of our analysis into the Liberal Politics Attention Index™ and the Conservative Politics Attention Index™.

Posted by dsifry at 10:31 AM | View blog reactions

RNC Wrap-up, Day 2

Arnold, the Bush sisters, voting machines, Kerry Campaign shakeup rumours, criticism of RNC bloggers, and censorship of Supreme Court decisions were on the minds of bloggers today as the second day of the Republican National Convention wrapped up.

  • Arnold's speech polarizing California governor Arnold Schwartzenegger provoked strong reactions from both Liberals and Conservatives. Lots of conservative bloggers swelled with pride at Arnold's speech, according to the Technorati Conservative Politics Attention Index™, such as Jay Reding, who wrote, "His story of living under the shadow of Communism is an important reminder of why America is still admired across the globe." Technorati's Liberal Politics Attention Index™ showed liberal bloggers reacting strongly to Arnold's jabs at Democrats: Luis Poza wrote of the Governor's speech, "full of sound and fury and signifying nothing".

  • The Bush sisters Jenna and Barbara Bush's speech provoked jeers from authoritative liberal bloggers, and Conservatives largely remained silent on the speech. Keith Berry wrote, "Up until now, I've never really felt bad for anybody with the last name Bush, but watching the Bush twins at the GOP Convention was was just so sad I was nearly brought to tears. It was a train-wreck. An honest to God train-wreck." On, a conservative blog, James K. Hat wrote, "This convention has been great so far. (Edit - great until the Bush daughters spoke... what in the world was that?)", and the Washingon Monthly rreports on other conservative reactions.

  • Revelations on an easy way to hack voting machines made their way through the blogosphere today, in this article on Bev Harris', noting that the Diebold GEMS central tabulator, used in many over 30 states, contains a stunning security hole.

  • Kerry campaign shakeup rumours Mickey Kaus reports on the rumours, starting in the Washington Prowler, that Kerry will shake up his campaign staff. This was later covered by MSNBC, and CNN.

  • Blogger's Corner events, and critiques General Tommy Franks came by the bloggers' area at the RNC for a photo opportunity, and while there announced his support for President bush, a prelude to his more official annoucenet on Sean Hannity's radio show. publishes a biting article on the actions of the RNC credentialed bloggers.

  • Redactions of Supreme Court decisions in Patriot Act suit?The Memory Hole reports on the "blacking out" of passages quoting US Supreme Court decisions in the ACLU's suit against the Justice department. This could be a breaking story in tomorrows news.

    Posted by dsifry at 2:41 AM | View blog reactions