Robot Walk-through

You’re all dressed up, and the hors d’oeuvres are on the table. But is there a big Do Not Enter sign on your door? You know the basics of how the robots find your site, and you know whether or not your landing pages are indexed. Today you’ll look for barriers that exist between the robots and your landing pages.

Take a look at your Rank Tracking Worksheet to determine whether any of your landing pages are not indexed. Here are several reasons a robot might not be reaching your landing page and possible ways to fix the problem:

Robots can’t follow your links. This could be as simple as having no links from your home page or your main site navigation to one or all of your landing pages. Or maybe the links to your landing pages are created using hard-to-follow code, such as JavaScript pull-down menus or pop-up scripts. Often, this is an easy fix: Just add standard HTML text links from anywhere on your home page to your landing page. (You’ll probably want a site map as well. We’ll cover that next month.)

Your site asks too much from the visitor. If the queen came to visit, you wouldn’t turn her away if she weren’t wearing the right hat. Treat your spiders the same way! Some websites won’t display to a viewer who doesn’t have JavaScript. Guess who doesn’t have JavaScript? The robots! Some websites require cookies. Guess who won’t accept cookies? You get the point. You’ll need to eliminate these requirements on your landing pages. If you’re not sure what your site requires, you’ll get a better sense of it when you look at the spider’s eye view of your website tomorrow. A server outage interrupted indexing. There’s nothing like that warm and fuzzy feeling your customers get when they encounter… an error message?

Perhaps your pages are linked and structured properly, but the robot came crawling just at the moment your system administrator spilled his Red Bull on the server. Either the robot captured an error message, or found no site at all to index. There’s nothing you can do in a situation like this but wait until the next indexing cycle. And if this seems to be a regular occurrence, look into a more reliable hosting situation. (By the way, for the perfect balance of caffeine and server protection, your sysadmin should switch to coffee with the little sippy lid.)

You told the robots to stay away. That wasn’t very nice of you! Later this week, we’ll get into the details of how you communicate with robots through a file on your site called robots.txt.

Your site is being penalized. It’s possible, but unlikely, that you are violating a search engine’s guidelines without knowing it. If none of the other problems are striking a chord and you are absolutely sure that your pages are not present in the index, and especially if you were ever engaged in questionable SEO practices in the past, this might be your situation. It’s a tough one. Probably your best strategy is to work through the rest of this week, make sure your site is squeaky clean, and use the search engines’ reinclusion requests. See yourseoplan.com for URLs.

What’s Popping Up at the Exploratorium

Lowell Robinson of the Exploratorium, San Francisco’s museum of science, art, and human perception, had a problem: how to display interactive video and Flash content on the museum’s Science of Gardening website (exploratorium.edu/gardening). Most of this rich, interactive content was built within separate pop-up windows. Lowell and his team knew that this could wreak havoc on their search engine listings.

Their primary concern was not that the search engines would have trouble indexing the pages, but rather that the pages would be indexed and site visitors would click directly to the pop-up content in a full browser window rather than a mini-pop-up window without entering the Exploratorium website:

“After putting hundreds of hours into producing this rich content,we didn’t want the search engines to index our pop-up windows as stand-alone web pages with no way to click to the parent website and none of our branding or credits displayed.”

The Science of Gardening team came up with a clever solution: The web developer placed a piece of code on the pop-up page that identifies the referring URL. Now visitors who are already viewing the Science of Gardening website see the regular pop-up (see the left screen shot below), while surfers coming in from search engines or other links see the Flash or video content wrapped in a proper branding package, with easy access to the rest of the site (see the right screen capture below). Although some less-than-ethical SEOs may use this method to trick the search engines and confuse site visitors (a technique called cloaking), this is one example of using the technique to help the audience rather than deceive them.

Lowell’s team learned an important lesson: You can’t always control how people arrive at your website! If you have a website with significant content displayed in pop-up windows or within frames, be sure to check how they look as stand-alone pages and make sure that you are comfortable with them being potential entry points into your site. If you aren’t, follow the Exploratorium’s lead and take steps to make those windows shine! (Full disclosure: Lowell is the beloved husband of one of the authors, and yeah, he got a little free advice on this project.)