Sifry's Alerts: State of the Blogosphere, February 2006 Part 2: Beyond Search

February 13, 2006

State of the Blogosphere, February 2006 Part 2: Beyond Search

In Part 1 of the State of the Blogosphere report, I covered the overall growth of the blogosphere. Today I'm going to cover the growth of the blogosphere as media, and discuss some of the emerging trends that deal with handling information overload. In a world of over 50,000 postings per hour, and over 70,000 new weblogs created each day, keeping on top of and in tune with the most interesting and influential people and topics is the new frontier beyond search. I've also got some surprises for you at the end of this post, two new features that I hope you'll find useful. But first, let's get our hands dirty in the data!

MSM vs. Blogs

To start, let's look at how attention has been shifting in the blogosphere. In the chart below, the top news and media sites are charted according to the number of bloggers linking to them, and clearly, people are still paying a lot of attention to mainstream media stalwarts like The New York Times, CNN, and The Washington Post.

For these sites, which sit on what I call The "Big Head" of the curve (as opposed to the now-famous "long tail", four blogs - BoingBoing, Engadget, PostSecret, and Daily Kos show up. This may look a bit smaller than the data of last August, but a quick look a bit further down the tail starts telling a more interesting story (Note that I've flipped the axes so that you can see more data):

As you continue down the media attention curve past the "big head", that the number of blogs starts to grow.

The Long Tail

The chart below shows the attention curve once you get past the blogs that look just like mainstream media above. It is important to note how long the long tail really is: this chart at this scale doesn't show it - the long tail of the blogging world goes out to 27.2 million blogs. To give a sense of scale, if this chart was kept to the same scale and I printed out the additional sheets necessary on regular 8.5 x 11 inch sheets of paper in landscape mode to show the entire long tail, the length of the complete graph would be about 120 pages long, making the entire chart about 110 feet long!

Movement along the curve

With so may blogs and bloggers out there, one might think that it is a lost cause for new bloggers to achieve any significant audience, that the power curve means that there's no more room left at the top of the "A-List".

Fortunately, the data shows that this isn't the case.

Thanks to the Wayback machine, here's a look at the Technorati Top 100 as it appeared on November 26, 2002 (bear with me if the wayback machine is slow). Then look at it as it appeared on December 5, 2003. And again on November 30, 2004. And again on April 1, 2005. And now look at it today.

Let's take a few examples. Have a look at PostSecret. It is the #3 site on the Technorati Top 100 today, with over 12,000 sites that have linked to it in the last 180 days. It didn't even exist on the chart in April of 2005. Or look at The Huffington Post. It is #5 on the Top 100. It too, didn't exist on the chart in April of 2005. Or look at the #47 blog in April, 2005 Baghdad Burning. This blog still is regularly posting, but has fallen to #304.

This should not be meant to imply that there are no network effects, or that a power law relationship doesn't exist in the Blogosphere. Of course there are network effects. But I want to go a level or two deeper than just thinking about the blogosphere as an A-List and The Long Tail -- for that's far too simplistic, and leaves out some of the most interesting blogs and bloggers out there.

The Magic Middle

This realm of publishing, which I call "The Magic Middle" of the attention curve, highlights some of the most interesting and influential bloggers and publishers that are often writing about topics that are topical or niche, like Chocolate and Zucchini on food, Wi-fi Net News on Wireless networking, TechCrunch on Internet Companies, Blogging Baby on parenting, Yarn Harlot on knitting, or Stereogum on music - these are blogs that are interesting, topical, and influential, and in some cases are radically changing the economics of trade publishing.

At Technorati, we define this to be the bloggers who have from 20-1000 other people linking to them. As the chart above shows, there are about 155,000 people who fit in this group. And what is so interesting to me is how interesting, exciting, informative, and witty these blogs often are. I've noticed that often these blogs are more topical or focused on a niche area, like gardening, knitting, nanotech, mp3s or journalism and a great way to find them has been through Blog Finder.

Explore: Dealing with Information Overload

Given that there's a lot of interesting topical posts by influential or authoritative bloggers in those topic areas, we formulated an idea: Why not use these authoritative bloggers as a new kind of editorial board? Watch what they do, what they post about, and what they link to as input to a new kind of display - a piece of media that showed you the most interesting posts and conversations that related to a topic area, like food, or technology, or politics, or PR. The idea is to use the bloggers that know the most about an area or topic to help spot the interesting trends that may never hit the "A-list". We call this new section Explore, and we've seeded it with some of the most interesting topics that we could find. But one of the nice things about Explore is that there are no gatekeepers, and that anyone who writes interesting topical blog posts can get included simply by tagging his blog and tagging his posts.

It's still pretty new, and occasionally an irrelevant post or two sneaks into the display. We're working on fixing that, but one of the new features we're launching today is the ability to subscribe to an RSS feed of any explore category, so you can now read the most interesting posts via your favorite newsreader.

These middle tier blogs also define communities of interest in the blogosphere. Its easy to think of the blogosphere as a cacophony of voices spread out over a big long tail distribution. But Blog Finder and Explore help resolve these thousands of blogs into topical, relevant communities of interest that interlink, refer to one another and often wrestle with ideas, discuss them and move them along. People often ask, "what blogs should I read?" And often times a good answer is, "you should read the posts from the leading blogs in topics that of interest you. Blog Finder and Explore make this possible for the first time on a wide variety of topics--- and in so doing we hope will the blgosphere more approachable, useful, and comprehensible to more people than ever before.

Filter By Authority: Giving YOU the power to tune your searches

There's one more big feature that I wanted to write about tonight, our new Filter By Authority feature. You can see this on all keyword search results pages, looking like this:

Explore

Clicking on the green slider allows you to easily refine your search results to show greater or fewer matching blog posts. For some searches, you might want to pick and choose only posts from blogs that have been around a while and are highly influential - so pick "a lot of authority" as shown above. I've found this great for searches on highly trafficked topics, like "George Bush" or Olympics, or on topics that are known to get a lot of spam, like mortgage or refinance. I find that it often helps me to also answer the question, "Who is the most influential blogger talking about XXX this week, and what did she say?"

Clicking lower on the slider gives you the ability to see how different levels of filtering affect your search results. For my ego feeds, I always want to see every single mention, so I turn off filtering for those feeds. I also love looking at the charts on the left-hand side of each search result to see what changes when I change the filter, too.

As we implemented this feature, we spent a lot of time thinking about how to name it. We frequently use the term authority on our site when we talk about inbound links, as in "a link is a vote of authority." So to maintain consistency we called this new feature, " sort by authority." But in no way should this imply a value judgment. More authority doesn't necessarily mean more good or more interesting. In many instances, less authority yields more interesting results: a greater diversity of opinion, less mainstream thinking, more individual voices. The authority filter is a tool to fine tune results, and its a great way to zoom in on the voices that are commanding the most attention, and then zoom back out and listen to the whole diverse medium that is the blogosphere. With so many voices we're happy to add a new tuning control!

This new feature is a beta feature, so we're looking for your feedback! Do you like it? Find it useful? Or is it confusing? What about the name? We tried a number of different names for the feature, but ended up picking "filter by authority" since we speak about a blogger's authority as being based on the number of links he gets from other people, but it isn't a perfect analogy. In the end, we decided that rather than having the perfect name, we'd much rather get the feature out there for all of you to try, and we'd listen intently to your feedback and comments.

In Summary

Blogging and Mainstream Media continue to share attention in blogger's and reader's minds, but bloggers are climbing higher on the "big head" of the attention curve, with some bloggers getting more attention than sites including Forbes, PBS, MTV, and the CBC.
Continuing down the attention curve, blogs take a more and more significant position as the economics of the mainstream publishing models make it cost prohibitive to build many nice sites and media
Bloggers are changing the economics of the trade magazine space, with strong entries covering WiFi, Gadgets, Internet, Photography, Music, and other nice topic areas, making it easier to thrive, even on less aggregate traffic.
There is a network effect in the Technorati Top 100 blogs, with a tendency to remain highly linked if the blogger continues to post regularly and with quality content.
Looking at the historical data shows that the inertia in the Top 100 is very low - in other words, the number of new blogs jumping to the top of the Top 100 as well as he blogs that have fallen out of the top 100 show that the network effect is relatively weak.
The Magic Middle is the 155,000 or so weblogs that have garnered between 20 and 1,000 inbound links. It is a realm of topical authority and significant posting and conversation within the blogosphere.
Technorati Explore is a new feature that uses the authoritative topical bloggers as a distributed editorial team, highlighting the most interesting blog posts and links in over 2,500 categories.
The new Filter By Authority slider makes it easy to refine a search and look for either a wider array of thoughts and opinions, or to narrow the search to only bloggers that have lots of other people linking to them. This gives you the power to decide how much filtering you want.

Technorati Tags: authority, beta, blogs, blogsearch, charts, explore, filter, msm, search, search engine, sotb, sotb2006, statistics, stats, weblog, weblogs

Posted by dsifry at February 13, 2006 7:32 PM | TrackBack | View blog reactions

Comments

David,

Great seeing you again at Northern Voice ... I feel like all of us got a sneak peak into this part 2 while there (like the Magic Middle). I also agree with you that the "A-list" certainly hasn't cornered the marketing on smarts, wit, or insight.

And BTW ... your link the Blogging Baby points to Techcrunch. ;-)

Posted by: Tris Hussey at February 13, 2006 9:08 PM

Another "Northern Voice" attendee here. I like the new "filter by authority" feature, and the thinking behind it. It's worth adding that it's in the "Magic Middle" that the most interesting discussions--in terms of comment threads and so on--take place. Obviously, these don't usually get searched by Technorati. But it's often where the Blogosphere is most lively.

Posted by: Jon at February 14, 2006 12:34 AM

Filter by authority is good, but it might be worth taking a leaf from Google's page, and make links worth more if they come from sites that themselves also have some authority.

Also, how about putting the tags in some sort of Yahoo! directory like category hierarchy?

Douglas

Posted by: Douglas Reay at February 14, 2006 4:43 AM

I like the idea of the new "authority" meter, and especially how you've implemented it. Rather than make users go to an "advanced search" form to enter a given threshold number of links, you just let them slide up and down between blogs with more and less incoming links.

I personally would just rename it to reflect the fact that all you're counting there is inbound links. But I can see why you just went with the word "authority" for the time being.

Posted by: Easton Ellsworth at February 14, 2006 6:28 AM

Hi Dave,

Another great report full of thought and bound to provoke conversation.

A question: As defined, doesn't the "authority meter" reinforce the status quo and favor the existing A-list? You mention that "inertia in the Top 100 is very low," but wouldn't that change if people were being directed/funneled towards those top 100 almost by default?

Keep it up. This is awesome.

~G~

Posted by: George Nimeh at February 14, 2006 8:48 AM

Filter by authority: The current slider prejudicially favors "authoritative" blogs by including them at all the lesser levels. I can't filter them out; I can only filter out the *less* authoritative blogs. Nice feature, but why not offer the option to see *only* posts with "little authority"? After all, as you point out, that's where some of the most interesting discussion happens.

It might be a silly idea, but just thought I'd mention it.

Posted by: Editor B at February 14, 2006 10:26 AM

I find the authority filter that David introduces much more interesting than Technorati's Blog Finder or Explore features. I fail to see how the latter are much different from regular tag or category searches, with results ranked by link popularity and chronology. - 'The Green Slider, a nifty piece of usability engineering' http://www.josschuurmans.com/josschuurmans/2006/02/the_green_slide.html

Posted by: Jos at February 14, 2006 9:24 PM

Fascinating insights, thanks Dave.

Posted by: Douglas Clifton at February 14, 2006 11:34 PM

I think the by authority meter is a good idea, if there are 5,000 hits for a term it's a great way to filter the results, giving us more choice then just 2 variables.

Since there are different settings you can filter for the A listers or the B or C listers if you're searching for a keyword in a category outside of the A list. It gives us "magic middle" people a chance to stand out in our own niches.

A third Northern Voice attendee

Pete

Posted by: Pete Quily at February 15, 2006 12:02 AM

Are you going to value the authoritativeness as well as the authority by factoring things like people who consistently post earlier or at more length? I'm still thinking about the calculus of attention we discussed...

Posted by: Mary Branscombe at February 15, 2006 5:33 AM

The new Explore feature is pretty cool, or at least it could be, if you indeed included all posts from "magic middle" blogs that tag in that category. Right now it seems that you have to have at least one link to your post and also be in the "magic middle" or above. Right now, PR and Public Relations is mostly populated by one "A-lister." Can this be adjusted?

Posted by: Kami Huyse at February 15, 2006 8:03 AM

would be great, if you could give some more clear explanations of the numbers shown in the graphs... i suppose it�s visits. what�s the source for the MSM visits you compared? the 'big head' graph i hardly readable, can you put up a larger image? thanks for those usefull basics!
et

Posted by: eyetag at February 15, 2006 8:51 AM

Great post, and so I almost hate to ask this, but I note that 33 is an odd number of sites to include in your blogs and MSM graph. What happens if you look at the top 50, or top 100? Do you still see about one blog per seven MSM link targets? At what point does that ratio start to change?

Posted by: Adam S at February 15, 2006 11:04 AM

The little green slider is cute, but if you want to be less cute and more functional, how about a standard popup showing several choices: 0, 1, 5, 20, 50, 100, 200, 500 or similar. You could also follow "editor"'s suggestion by providing two standard popups representing an upper and lower bound.

Also, I continue to be baffled why you removed the technorati rank which had previously been listed in text in each search result entry. Instead, one has to mouse over an icon, and that occasionally doesn't work. Frankly, I have no idea why anyone would do that.

Posted by: PYT-LA at February 16, 2006 12:53 AM

"The Wayback machine"?!? yes, I was giggling at that one...but the Magic Middle is really the most fascinating thing here. Who'd have thought this would happen? Being part of that Middle, what goes on is a combination of searching one another, commenting and linking. From my vantage point, it's pretty social--and the social part of it counts as much as the information dissemination part.

Oh, and H.P. is a ringer in the stats. Easy to move up when the blogger, and most of the people blogging on the network, are marquee names. Post Secret is a bit better example of going from Zero to Hero in no time--demonstrates that uniqueness of information presented can indeed impact one's popularity (although one might have to think pretty hard to come up with an idea as unique as Post Secret.)

Posted by: Tish Grier at February 16, 2006 5:30 AM

I would like to know if is there similar report or study about podcating or podcaster?

Posted by: Carlos Toledo at February 17, 2006 12:29 AM