Crouching JavaScript Hidden Intelligence

A Look at JavaScript Code in Relation to Search

JavaScript and the Engines

It is now pretty much common knowledge amongst Search Marketers and Developers that JavaScript is ignored by Search Engine Bots, effectively keeping any code or content within JavaScript tags out of the Search Engine Indexes.

The reason behind this is that bots are, and have been, built to be simple creatures by nature, unable to crawl more advanced code, and other rich content such as images and flash. Perhaps this was partly done to save on processing and memory usage, though it can be imagined that indexing the entire Internet raises some tough logistical questions that need to be worked out, to ensure a stable environment. The “Big 3″ or Tier 1 Search Engine companies have Datacenters the size of football fields, many times over, but before we continue in such broad strokes, how about a small test.

The Document Title JavaScript Test

A few months ago I launched a page to test how JavaScript is handled by Bots. The working assumption was of course, that the JavaScript code would be ignored. I decided to test the JavaScript Document Title against the standard title tag that you would normally see in the head of any HTML formatted page. Sure enough, the result proved that the JavaScript command was completely ignored, and the text within the title tag was used.

Looking at this test meta page result in Google, the title tag is displayed.

However, the JavaScript command does overwrite the title tag in the browser title bar.

This doesn’t just prove that JavaScript is not crawled by bots, this also makes a case for content that is displayed to a user but hidden to Search Engines, in other words an Invisible Web.

Invisible Web

The term “Invisible Web” is used to describe all that is not found in the Search Engines, and sites that are coded heavily in JavaScript can be defined in this way. A post from Eric Ward’s URLWire from July 2002 gave this  announcement: A New Search Engine Tool Searches Javascript Files, a tool that still exists today. As Search Engine Algorithms become more advanced, the technology that supports these algos will become more advanced too. However almost seven years later, the major Search Companies have only begun to start digging into parts unknown, such as forms and flash. On the Internet map of 2009, there (still) be dragons.

JavaScript, the precursor to Nofollow

JavaScript has been used in the same way that the nofollow attribute is used today, to prevent sending link juice between pages. Between the two, JavaScript is still the proven way to do this, whereas nofollow may send some value between pages, but this is not yet proven. JavaScript has also been used for cloaking methods, and for more malicious purposes.  

Practical Uses for JS

The benefits of being able to display different content to a user versus the engines has merit, and there are benefits. However there are clear lines and limitations to how far this can be taken. The Search Engines prefer that you serve them content as you would a user. Do not shy away from coding in JavaScript for the sake of the Search Engines. If it makes sense, and there are benefits, and they fall within best practices, then code away!  On an application level, JavaScript is used by many companies, including the Engines for their recent software, such as MSN’s adCenter Analytics, Google Analytics, Google Chrome, and Google Optimizer to name but a few.

It is amazing to think that bots wouldn’t exist without JavaScript, and yet they are unable to see it, even though it’s all around them as they scour the web!

CMA