I recently decided to take a look at the Sitemap.xml protocol, and see what all the fuss was about. I’d always assumed that it was one of those things which doesn’t apply to me, and stuck my head in the sand. What I wasn’t ready for was the huge can of worms that my investigations opened.

‘Sitemaps’ are basically just xml lists of web pages, which instruct search engine robots on what to crawl. Their purpose is to make life easy for search engine robots, and enable them to easily find everything on your web site—including stuff that isn’t linked properly.

Firstly, creating a sitemap xml file isn’t straightforward. Google provides a Python script for this, but this was one learning curve I wasn’t prepared to take. So, I looked into alternatives: Dreamweaver can produce Google sitemaps with the aid of an extension, but this is no use for some of the dynamically generated links on my web sites; similarly, Wordpress plug-ins are available to generate sitemaps of permalinks, but this isn’t much use for other content. After a bit of detective work, I came across a program called gSiteCrawler, which is a Windows application which behaves just like a finely configurable search robot.

Okay, now you’ve got your sitemap.xml file, you can upload it to your site root and job’s a good ‘un? Not quite. It was at this stage I realised just what search engines have been indexing all these months…

Close inspection of my web site’s statistics in Google’s Webmaster Tools microsite revealed scores of duplicate pages which I thought I’d removed a few weeks ago, using a Wordpress theme modification (which generates meta robots=noindex tags on the fly). This galled me, because I thought I knew all about Wordpress’s duplicate content weaknesses and the solutions. GSiteCrawler’s sitemap also showed further duplicates which I was unaware of.

There then followed a few days of severe head-banging-on-wall with Webmaster Tools, before I realised four things: (1) Wordpress’s 404 page actually gives a 200 (OK) and needs fixing so it gives a proper 404 (not found); (2) Google’s ‘Supplemental Results’ are untouchable — don’t for a minute think you can change them; (3) you can’t remove anything from their index unless you’ve made sure your robots.txt file backs it up; and (4) ‘duplicate content’ extends to pages you couldn’t be bothered to give custom titles and meta description tags.

Don’t get me started on robots.txt. The syntax for this file is voodoo, and very poorly documented. I ended up changing my directory structure to be as simple as possible so that robots.txt could screen duplicate content without much effort. Trying to be clever with robots.txt was a waste of time and brain cells. But, used well, it can stop robots scanning pages not needed to be seen by the search engines, and save them the time and effort of indexing them—time better spent spidering the rest of your web site.

So, the moral of the story is that using a CMS to generate pages, and/or being slapdash with title and meta tags, and/or lack of attention to ‘duplicate pages’ seen through alternate URLs can cause a hell of a lot of nightmares, which all need addressing before you go anywhere near a Sitemap and Google’s Webmaster Tools.