Report Says Ajax Monkeys with Search Crawlers

Ajax not only disrupts the entire page view model, but apparently it monkeys with Google's crawlers too.

According to a report, the reader content that's being added to the new USAToday.com is not going  not be indexed by Google or other search engines. That's because all of the new goodies, including reader blogs and other social-networking and bookmarking features, were built with Ajax. Man all that potential Google Juice is going to waste.

Is this true? It's news to me but it seems logical. I wonder if this will slow down the move towards building sites with Ajax. SEO jockeys, please weigh in.

Comments

[Matt Harwood]

I can't see the issue, unless the USAToday site does not have a 'permanent' / static URL to content. If this is the case, then SOL in my opinion (unless google spiders the site every minute!).

[Daniel R]

The social networking/profile pages portions of the USA Today website are behind JavaScript-called URLs and AJAX-based pages. It will probably hurt USA Today in the long run, but the bulk of the web site, as it is now, like the articles etc are not effected.

Basically, the issue with AJAX is that the user clicks on a link, sees the different content, but the URL is the same.

Check out Google Maps, even after you type in an address or drag around the map, the URL stays the same.

Search engines need to see 1 unique URL for 1 unique content. With AJAX you can get tons of different content and even have the user click on things to see different content, yet stay on the same URL.

There are workarounds possible for this, but they need to be planned into the backend as part of development.

[ted]

From what I see, they should be fine. There's no real valuable user-gen content trapped in AJAX. Comments and basic profiles have their own dedicated pages.

[Kyle]

Hi, Steve -

I've been attempting to dispel some of these SEO myths lately, so I'll join in with a quick comment.

The report raises a good point, but it is still off the mark. AJAX is not the fault here, if USAToday can't be indexed by Google and others, it's because of the specific way in which they implemented AJAX.

I'm not sure why, but there seems to be a belief amongst SEO experts that you can't use AJAX at all. In reality, you can use all of the AJAX you'd like, and still have it be SEO friendly. It really boils down to the developers implementation of AJAX. If the developer has used modern best practices of progressive enhancement to make sure their CSS, HTML, and JavaScript all live in separate files, then the page should degrade gracefully, and still be accessible to people without JavaScript, as well as search bots who can't manipulate JavaScript.

For a clear example of this, where I work at Revolution Health, our Insurance product uses a great amount of AJAX to return insurance plans and, if the user has JavaScript enabled, is never directed to another page to view individual plans. However, if user doesn't have JavaScript, the page will degrade gracefully with a full page refresh and direct the user to another page with the same information, indexable by Google.

Try it out for yourself at: www.revolutionhealth.com/insurance/

Anyway, my apologies for the long post. As I said, I've been working recently to dispel the idea that AJAX means your pages can't be SEO friendly. Somebody needs to educate SEO experts, and judging by implementation of Javascript on the USA Today, some web developers, on modern web development best practices.

[Jarid]

To put it simply, crawlers don't execute javascript. So, if you build a page where the content is displayed asynchronously using AJAX (i.e. "loading..."), you also need to build a synchronous version of the page in plain old HTML for clients that don't support javascript (e.g. crawlers).

[solay]

I buy that its true, hard to believe though. I would not reccomend ajax because of this.

[volkan.ozcelik]

I *totally* agree with ted.

I also have a pure ajax beta startup which displays almost nothing when JavaScript is disabled.
But it's relatively easy for us to make our web application google-friendly.

Because we have architected it keeping this fact in mind from the beginning -- and we plan to make the site totally accessible when JS is turned off before we get out of beta.

(why we did not start with a plain old html page and then enhance it with AJAX -- i.e. why we did not follow the best practices -- is totally another story)

Jarid,
Have you googled for "progressive enhancement" ?
It's perfectly possible to use AJAX and be search engine friendly. Or did I misunderstand your statement?

Cheers.

[volkan.ozcelik]

just a redux to my former post:

replace "I totally agree with Ted"
with "I totally agree with Kyle"

Ted, I also agree with you btw :)

[Mike Levin of HitTail]

When we programmed HitTail, we accepted that any content delivered by Ajax would not be visible to search. This is the age-old issue. Is your website going to work like an electronic brochure (most of the Web) or a program? If it's going to work like a program, displaying variable data based on conditions, you can't really expect it to be included, because what displays is based on the unique actions of that particular user (without a page reload).

The graceful degradation mentioned by Kyle is good if you've got a brochure-like website with some programmatic features, and that appears to be the case with USA Today. But if it's all program (like a shipping app), then forget it. The solution there is to (unfortunately) break out the search friendly requirements into a separate venture. But that makes sense too, because who wants a feature of your app to come up in search? There would be a potentially unlimited variety of each page.