Automate xml Sitemaps

Automate XML Sitemaps – In the past you had to create several version of your sitemap for the different search engine bots. They required these to properly crawl your website's content for indexing (inclusion in their results)

Since then, two major changes have been made.

1. A universal sitemap format was adopted: xml (this even includes Ask.com)
   
2. A tweak was added that tells the bots to go to your robots.txt file first and look for a path to the xml file so that it knows where to go, and additional features that allow you to prevent the bots from crawling and indexing unnecessary files such as cpanel, administration or even image files.
   

You can specify the location of the Sitemap using a robots.txt file by simply adding the following line:

Sitemap: <sitemap_location>

The <sitemap_location> should be the complete URL to the Sitemap, such as: http://www.example.com/sitemap.xml

This directive is independent of the user-agent line, so it doesn't matter where you place it in your file. If you have a Sitemap index file, you can include the location of just that file. You don't need to list each individual Sitemap listed in the index file.

There are lots of places that have a free xml sitemap generator;
SourceForge.net
xml-sitemaps.com
auditmypc.com

I think GSoft also has an Open Source program that will automatically create the xml sitemap and upload it via your ftp if you set it up.

Because of the ever-changing content of a properly optimised site, as well as sites with CMS's (content management systems) and the millions of static sites out there, this is the method that I recommend.

Additionally, fresh content will keep the robots coming back to index your site. Most of these programs will insert the creation date in to the file to document that it is a revised version, but the most important part is that whenever you change anything within the site (and based on this article you may have a few changes to make) this will assure you that it will be picked up by the engines automatically without having to spend the time that we did in the past to expedite this process.

Part of the problem is that if you add navigation to this new content and put it in the site template, as I mentioned earlier, search engines will parse (remember) the content and skip over it to preserve its allotment of data that it can crawl on each url. They want to crawl deep and get as much content as possible, so they skip pages that provide no new content.