Search is Broken

Search is broken. You may not feel the same way, but it is. And it might require a new web standard to emerge in order for it to truly be fixed.

Now with Google at over $600 a share and 91% of us using search engines (and largely satisfied with them) it's easy to believe that search is anything but broken. However, the problem is that the entire search paradigm was built during the Web 1.0 era when generally all we cared about searching was the content that other people created. That's not true anymore.

Today all of us are generating a torrent of content across lots of different sites - sometimes hundreds of them. This includes everything from financial transactions your bank site is logging to email that sits in your Gmail or Hotmail account to content you're posting on Facebook and, increasingly, documents you're creating using rich internet applications like Google Docs. In addition, lots of data comes back at you too - replies, account alerts, pokes, etc. There's currently no way to easily search it all.

Right now when you want to find bits that you generated you have to think first about where it lives and then go to that site and search for it. Here are some of the search boxes that I utilize on a daily basis. You may recognize them too.

Now, I don't know about you but I don't want to think about where my information lives or use 100 different search boxes. That's the beauty of Google. You go to a single box you enter what you want and instantaneously you find it. You don't have to think about what site it's on. Google knows so you don't have to. We need a secure, opt-in web version of Google Desktop that can find all the bits we generate online, even if they are behind walls.

At a minimum, the portals will evolve so that you can not only easily search the web, but all of the stuff you generate on their properties via one box. You should be able to go to Google.com and search Gmail, GCal, GDocs, GReader and Jaiku. Right now you can't - easily.

However, with a little bit of collaboration, the opportunity is much bigger. It would be great to see a secure XML standard emerge for data sharing. This way, if I enter a search in AOL.com and I let it access my Gmail account, it will pull up my messages in the search perhaps in a special tab. This is a topic I am hoping some of the speakers will address at next week's Search Marketing Expo, which I plan to attend.

I will let the technologists figure the details out. What I do know is that people don't want to think about where their data is. In my case, I have upped my Gmail storage to 30 gigs because I use it as a private intranet (I have written about this extensively). I constantly email stuff into it - e.g. web pages, personal docs, etc. - to basically create the one box interface I am looking for. Others do the same or use Greasemonkey to fuse sites together. But that's not what normal people want.

So search is broken. If we leave it to Google they will fix it - but only within Google's land. How about Yahoo or Microsoft? Same. They can only search their own servers. It would be great to see everyone work together to make everything an individual wants to find searchable from one place.

Comments

[Nathan Gilliatt]

Steve,

This raises some rather obvious privacy and data security issues. Using a Google service as a "private" anything requires placing substantial trust in Google's (a) commitment and (b) ability to maintain the privacy and security of your information (permanently, not just this month). Do you really want to plug your financial accounts into the Universal Data Sink?

(Not to pick on Google. The same concerns apply to any entity, public or private, that would aggregate so much data.)

Spend a little time reading the Risks Digest ( http://catless.ncl.ac.uk/risks ). That will increase your skepticism about the reliability and security of computer systems.

[Edward Vielmetti]

Steve,

The librarians of the world call this "federated search". One search box, but it searches multiple databases simultaneously, with search results sorted in some reasonable way.

Amazon had a product (A9) that did this, but they canned it. Microsoft had a technology for returning search results in a way that was easy to federate (Open Search), but that also didn't take off.

IFTF had a post about this a few years ago:

http://future.iftf.org/2005/01/liquid_informat.html

I handle this personally with Firefox's search selector box - depending on the need I search any of 24 search engines from a single box (including two or three library book finding systems that are more relevant to me than to anyone else).

[Steve Rubel]

I personally don't take Gmail as far as to put stuff in there that shouldn't be. However, at minimum you should be able to search all of your Google bits from one box, no?

[Steve Rubel]

Ed, you still need to remember where something is, even with Firefox keywords or the search bar.

[Roger L. Cauvin]

Google Desktop search accomplishes much of what you are describing. When you have it installed, you have a single "tabbed" web page or query box from which you can search e-mail, calendar appointments, contacts, IM, documents, the web, newsgroups, etc.

[Robert Young]

Steve,

they actually have a limited thing of the multi-search service out. aftervote.com . It searches Google, Yahoo, and MSN.

[Steve Rubel]

Roger, Google desktop is a good start I agree. Does it integrate with GCal and GReader or does it only search outlook?

Robert, thanks but that's still all web based content, not your own bits.

[Todd Defren]

I certainly empathize with the sentiments laid out here, Steve. Ever since I downloaded MSN Windows Desktop Search to my PC (Google Desktop Search slowed me down much more), I completely forgot about "filing" my documents in special folders. Just "search the desktop (and/or network servers) and voila, up comes the content.

To be able to extend this experience across all the content & conversations generated across my various "lifestreams" would be awesome.

[ekivemark]

Great thought provoking stuff! This is an area where a meta search engine could provide valuable differentiation and great user convenience in bringing together our various accounts into one search box.

[Simone]

Steve, cool post but I don't think that search is broken as much as the social networks / blogs / etc are broken. You're talking technology, my point is instead that what you don't want is HAVING YOUR BITS SPLITTED IN TENS OF PLACES in the first instance. What we need is a simple to explain / easy to handle aggregator. From there the search thing is easy.
http://www.harael.com/2007/09/google-should-build-on-long-tail-of.html
If your point is that search engines could act as an aggregator themselves, so nobody's going to FB/Twitter etc anymore but looks for content on Google, then we're probably saying the same thing.
Ciao
Simone

[Lars Teigen]

This is part of the problem that Second Brain is working to solve.

We want to become a place for all your internet content. Add the services you're using and all your content will be synchronized automatically with your Second Brain library, making you able to search and browse in everything from a single interface. You can also organize your content in collections and show your recent internet activity on your public profile page.

It's a big idea, and we have lots of work left, but we have an early beta preview live on http://secondbrain.com and we're very curious about any feedback you guys might have.

[Mark]

have you seen lijit.com? It's something I use and love on my website for indexing my content, utilizing google's algorithm, but searching only my content. You think it up once as to where your content lives, and it finds it.

[Bill Burke]

I agree, and disagree with a lot of what's commented, above. I _really_ like Lar's "2nd Brain:" appliance, above. Excellent model, Lars, and I hope your idea(s) work out.

I'm kind of an "iMedia Fan".. they typically have good info. on marketing and stratagies thereof.
Here's a nice article on "Paid Search"..
http://www.imediaconnection.com/content/16870.asp

Bill Burke
http://www.adondo.net

.

[dc crowley]

Very good post Steve. Trust and privacy issues aside - and from a Google perspective, jaiku has a lifestream aspect which might be useful but I save pdf's and some non-documents on gmail and documents on Google docs, photos on Flickr... finding stuff that is lying about everywhere including my computer is a challenge. Our order of today will be obsolete in 2 years time. The way we interacted on internet 24 months ago is not like we do today. Search, like Mac OSX spotlight or google desktop do it on the computer. You are right. We need it for the web. Google need to create a secure rss/ atom distribution - it is a major issue. It should not matter where stuff is. I should just be able to find it and "I can't". Google desktop is mot the answer IMHO, Google webtop... maybe. I work on different computers, different locations.

[Yen]

Steve,

Having come out of Yahoo's search division, I agree.

The pain of searching across my docs, social networks, email etc. is real enough. Even with the help of X1 (I use the yahoo desktop search version), it's time consuming and sub-optimal. I would love a solution that searched based on the underlying semantics of emails, photos etc.

Interesting you focused more on the pain of searching the bits of content we individually generate rather then searching other people's bits of content. But I have heard numbers thrown around that 95%+ of content is "off web", on hard drives etc. - and generally inaccessible

At Kango, we have tried