Turning the tables on Total Information Awareness

Aaron Swartz writesabout John Gillmore's response to a SFWeekly article turning the tables on Poindexter's Total Information Awareness program. Gilmore's challenge? Find out everything in the public record about Poindexter, post it to the web, and publish it.

Ouch! Best use of comedic irony I've seen all month.

I don't think this kind of campaign should be done to anyone, but perhaps this will help to open people's eyes about what this frightening big-brother proposal can do to all of us. And it is pleasantly ironic that the person whose privacy is about to be invaded is the one who will run the program that can do it to us.

Long and short? This is an outrageous law, and it should be changed. We must wake up and remember that freedom and liberty are the very things that make America great, and are the things that we must hold most dear, because otherwise, the terrorists have accomplished their goal.

Hello Police State?

Sorry, I'm going to get off of my techie soapbox for a moment, and talk a bit about world events.  If you're not interested, keep clicking on to other stuff.

Dan Gillmor is dead on, yet again:

Washington Post: In Terror War, 2nd Track for Suspects. The Bush administration is developing a parallel legal system in which terrorism suspects -- U.S. citizens and noncitizens alike -- may be investigated, jailed, interrogated, tried and punished without legal protections guaranteed by the ordinary system.

If you want to see America turn into a police state, just go on about your daily life as if nothing has changed. After all, your silent assent is what Bush and his constitutional wrecking crew expect from the American public.

If the executive branch has the unilateral authority to invalidate your legal protections, the Constitution is basically dead. And if you think that this power won't be abused more and more over time, you have never read any history.

It's frightening, really - is this how fascism took over seemingly constitutional democracies in the early twentieth century?  Don't forget - those people were elected into office, too - and kept the general populace complacent because they were only going after the "others" - the Jews, the Gypsys, the homosexuals, not the good, upstanding people who had nothing to fear, of course, as long as they toes the party line.  The "Total Information Awareness" program in particular scares me, for so many reasons, and as a computer professional who has been around the block a few times, the risks of abuse alone are frightening to consider. 

I think that the only way to fight terrorism, I mean really fight terrorism at its root causes is to truly act as we preach - if we really believe in democracy and transparency , then let's practice it ourselves.  If we really believe in equality, then let's put our money where our mouths are, and follow through.  Let's support democratic elections and systems, even when the new rulers don't align with our short-term interests.  Let's stop supporting those who oppose liberty, tolerance, and equality, like the various kingdoms and dictatorships that we currently support out of "necessity".  The American people are looking for a vision of hope, and a hopeful vision of the future, but what they're hearing is fear and hate and surveillance, and the "opposition" party is sitting on its hands. 

The world is a big and scary place - yes.  It's time to start practicing what we preach and not acting like the world's bully, because the way to true security is through more liberty, both here and abroad.  With our current actions, we only further incense hatred, and make things worse.

Technorati

I wanted to let you know about a new project I've been working on called Technorati.  Essentially, it is a site that creates a set of web services that I've always wanted for myself - services layered on top of the wealth of current search functionality and tools available for bloggers.

There are 4 core services that Technorati provides:

1) Link Cosmos: This is a service that lets you see what blogs are linking to a Blog (or any arbitrary URL, btw).  I always wanted this because I wanted something to help me get a feeling for how interesting a blog was - if lots of people are linking somewhere, it must be interesting, right?  And If lots of people I know and respect are linking to someone or something, then it must be something worth taking seriously.

So, with Link Cosmos, you can check any URL you like - type it into the search box, and you'll get an instant view of the bloggers who are linking to the site.  This is what makes it significantly different from Blogback, for example.

What I found out when I started populating my link database (it has over 1 Million links in it right now, and it's only been going for about a week) is that I would see an interesting URL or blog, and then recursively follow the Cosmos for the URL - things turned into a '6 degrees of separation' kind of fascination as well.

You can try out the Technorati Cosmos links at one of two places: Either the Cosmos link page, or at the Technorati Top 100, a listing of the top 100 blogs based on the number of other blogs who link to them.

If you find an interesting new blog in the top 100 - simply click on the (Cosmos)  link and you can see who is linking to you.  Click on their cosmos, and see who is linking to them.

This brings me to the other interesting thing about the Link Cosmos - you can see links to your blog that don't even show up in referer logs.  There are a lot of blogs out there that aren't frequented very often, and you won't see them in a site referer log unless someone actually clicked on the link to your blog.  It's a neat way to find new and interesting sites that haven't been "discovered" yet.

2) Google Rank

I have always been interested in how Google works, and also how I rate in relation to certain search terms, so I created Google Rank and Google Juice (explained later).  Google Rank allows you to type in a search term, and it then shows you the top 100 sites on Google for that search term.  The interesting part is that Technorati also goes back day after day and rechecks Google's rankings for that search term, so you can see how Google's rankings change over time.  Here's an example of Google Rank in action, for the ranking of "weblogs", for example.  The numbers next to the rank show how the ! rank has shifted over time.

3) Google Juice

A corollary of Google Rank is Google Juice.  By adding a URL to search for, Google juice gives me the ranking for a site (or set of sites) that I want to track - and Google Juice goes even deeper than Google Rank - if your site is in the top 1000 Google responses, Google Juice will find your site and give you its ranking for that set of search terms.  Here's an example, searching for the URL "www.scripting.com" on the search term "weblogs".  Rock on, Dave!

4) Watchlists

Watchlists are the easy way to keep and track historical information about the sites that you are tracking.  They allow you to quickly see new links to your site or see historical Google ranking information not shown in the standard views.

Watchlists are where the business model behind Technorati comes into play.  By paying $5/year, you can purchase access to the watchlist web service.  You get instant access to the live watchlist information, and you also get a daily email with all of the updates to the sites or Google Rankings that you're tracking.

One of the things that I find interesting is this business model experiment - will people find enough value in the Cosmos, Google Juice, and Google Rank services that they would be willing to pay a nominal amount for automated alerts and advanced features?  And what would be the appropriate price?

I hope that you have a chance to go to the site, give it a run through, and give me your feedback. 

Google web results significantly different from API

I've been playing around again with the Google API tools and I've noticed that they're using an old view of Google's index.  I probably wouldn't have noticed this if Google hadn't made such radical changes recently to its search results. 

Anybody got any good Google HTML parsers that don't use the API?  It's unfortunate that I have to ask for this, but if the results are going to be different, I want to make sure that I'm looking at the results most humans will see.

This is a major disappointment, because this really destroys the utility of a Web Services API - if the same query returns different results depending on the input method, we're put into the position of parsing through HTML in order to get the "real" results.

Perhaps this will change when Google comes out of Beta period with the API.  When they do, I'll also be interested in understanding their terms of use of the API for commercial purposes as well.  As it is, the Terms of Service say that we can't use the API calls for commercial purposes without written permission.  Bummer!  I've sent an email to api-support@google.com requesting clarification.  The folks at Google are smart and have good heads on their shoulders, my hope is that they do the right thing.  Anyone at Google listening?!?

Multihome sweet Multihome

I've just completed the server move from their old home in the basement of VA Software to their new home in Linuxcare's brand-new multihomed data center.  The servers are now physically and topologically closer to the office, and the throughput is better.  And another plus, these machines are now plugged into two different net connections, provided by different providers, and they are load balancing the connections with linux's new iproute2 package. 

Werbach's Supernova

supernova_button.gifKevin Werbach's new conference, Supernova, is happening in Palo Alto on December 9-10, and I'll be there as a speaker.  I'm also looking forward to meeting the other speakers, including a bunch of people that I've never met, like Dave Winer, Cory Doctorow, Jeremy Allaire, Rob Glaser, Mike Helfrich, and others, like the inimitable Doc Searls, who I've known for some time.  I'm sure that the attendees will be very interesting and smart people as well, given that the topic of the conference is decentralization, and Kevin's interests in open spectrum, weblogs, WiFi, and Web services, some very interesting topics today.