How to Data Mine Google Reader Feeds for Trends

Although I was never the A+ math student in school, I am a big fan of drawing insights from statistics - if the method is simple. If it involves math that ends in "ometry" then it's way over my head.

In addition, I am also a huge believer in studying tendencies. Humans are all creatures of habit. Identify someone or some group's patterns and you can figure out directionally where's they are headed. This makes it easier to spot and capitalize on trends no matter what your interests are. (Believe it or not these lessons come from reading about NFL coaches who actively study player/team patterns "on film.")

Google Reader - my favorite RSS application - recently added a powerful search functionality that has made me infinitely better at studying people and their social patterns. Using Google Reader you can now search an individual feed, tag or a folder and get back a total item count, all sorted by date for as long as you have been a subscriber to that feed. In my case, some of my feeds go back to October 2005 when the Reader first launched. That's a ton of data to mine for trends.

Now that my reader shows a huge cache of posts, I am subscribing to tons more feeds, stuffing them into a folder solely for the purpose of data mining them. The site also has limited set of advanced search operators. One hopes they will add more. It's worth noting that I don't actually read these high volume feeds. Rather, I mark them all as read so they get logged in my feed database and can be searched for insights.

Let's take a look at this in action at a very simplistic level. One of my favorite blogs is Lost Remote. I have been subscribed to their feed ever since I started using Google Reader. So I have two years worth of posting data to mine.

Let's take a look at some searches for the major TV nets and the results they returned.

Let's take another simplistic example. Is Robert Scoble's showing more blog love to Facebook and Twitter than his newborn son, Milan? Hmmm, the data shows it. (Just kidding Robert!) This is just a superficial analysis of his blog but in reality I could also add Robert's Twitter stream do the same run as long as it all lives in a Google Reader folder.

There's much more data here than what I have in the chart. When you actually look at the search results, patterns emerge. The vast majority of Robert's Facebook mentions came after they opened up their development platform in May. He only mentioned the site 14 times in 2006. Now imagine I ran this same search across all of the big tech bloggers, the digg home page and Techmeme feeds - all at once. What would I learn? Data breeds insights. And insights makes you smarter at whatever you want to accomplish.

A lot of the very basic stuff - e.g. searches within a feed - you can glean from using Google Blog Search, Blogpulse and Technorati. However, do not underestimate Google Reader. If you subscribe to feeds just for the sake of data mining and organize them the right way, you will be able to read tea leaves better than you can using a search engine. This will make you smarter at whatever subject you want to follow. It works best on full text feeds, but try it on mainstream newsfeeds too. You can learn a lot about what words make it into headlines and how often.