Step Wise SEO

With a little sleuthing, you can determine which search engine robots have visited your site and when.

To scope out robot visits to your site, do the following:

• First, find the name of the robot that you are interested in monitoring. Table 1 lists the robot identifiers for the four search engines, but be prepared for some variations.

Table 1 Robot Identifiers for the Four Major Search Engines

Search Engine Robot Name
Ask Teoma
Google Googlebot
MSN msnbot
Yahoo! Slurp

• Then, review your website stats program. Look for a section called “User Agents,” “Robots,” or “Browsers” (servers interpret a search engine robot as a type of browser).

• Depending on the sophistication of your stats program, you may be able to specify a date range and view robot visits. You might even be able to see exactly which pages within your site were visited.

If this all feels a bit too tedious, you may wish to look into software that provides simple yet detailed reports on robot visits. One such program is Robot Manager, available here: www.websitemanagementtools.com.

Keywords: Sitemap, Google Sitemaps, indexing,