February 2006 Newsletter

Site Maps

**** The 4specs Perspective

Are all your pages included in Google?

Unfortunately not all website pages are included in the major search engines. This is primarily due to home pages that use images and javascript for links. In addition, some content management systems create search engine problems. Both problems can be resolved by adding a sitemap - as described in this article.

If all of your important web pages are not included, it could cost you 3,000 users per month from just Google. This was the subject of a newsletter about two years ago - Get all the users you can for FREE.

This month's newsletter covers the use of site maps, including a new program from Google. The cost to implement step 1 should be under 2 hours and is well worthwhile.

1. Site Maps - put a text link on the bottom of your home page to a page with up to 100 pages you want to ensure are in the search engine's database. Call the link "Sitemap" in text - not an image. It is that simple. One 4specs' advertiser had an increase in 3,000 users per month (yes - every month!) from Google simply by adding their sitemap. This first step will work for Yahoo, Search.MSN and other general search engines as well as Google.

2. Google Site Map - Google has a new Sitemap program that Google will use to index your entire website - even sites over 50,000 pages. So far Yahoo and Search.MSN are not using the Google sitemap.

The easiest way to start developing a Google sitemap is to use the [site no longer works] sitemap generator. Be sure to scrutinize your sitemap carefully before submitting. Be certain that every page is included and pages you don't want included aren't included. Place the resulting XML page on your server and notify Google the site map is there. Update the sitemap page whenever changes are made to your site.

Most construction product websites are fairly static and not often updated. We frequently change 20-50 pages per day on 4specs by adding new ads and new listings and deleting or changing listings that no longer work. The 4specs sitemap is downloaded every day by Google and ensures the changed pages are updated.

After adding the AuditMyPC sitemap, we changed to the Google sitemap generator, running it on a nightly basis. This is more complicated as it runs in Python (a computer language found on most UNIX servers) and took me 3-4 hours to get the configuration file setup and working as expected. The AuditMyPC map took less time to initially get started, but requires a manual operation to run and upload the sitemap after each website update.

3. Google Sitemap Benefits - Several days after submitting the Google sitemap (either method), you can visit the Google page and identify the following:

• The top 20 search terms at Google where your pages are listed. This will give you an idea of search terms to include in your optimization plans. There is no other way to get this information.
• The top 20 search terms people clicked on to visit your site. This information is also available from your stats reports.
• Errors in downloading pages and other problems. This may be the only way to determine whether your website has problems from the search engine's perspective. These problems include HTTP errors, unreachable URLs, URLs restricted by robots.txt, URLs not followed, and URLs that timed out when the search engine went to your site.

4. House Cleaning - The next step after the Google sitemap is up and running is to clean up problems in your site reported by Google. This can be time consuming. I believe that your search results will be better if the search engine records few problems from your site - i.e. your website cleanliness increases the credibility of your site from the search engine's perspective.

Here are the key tasks you can expect to do in the house cleaning:

• robots.txt - Is this file accurate and up to date. Robots.txt is a text file placed on your server that informs robots (the programs used by search engines to traverse the web) which files you do not want indexed. Polite robots respect this file. For example, these newsletters are excluded from indexing by the robots.txt file as I do not want calls from people outside the construction industry. [added later] There is a new protocol where you can list your site map in your robots.txt file - Google, Yahoo and MSN will also find the file there. The top line in the 4specs robots.txt file is looks like:
  Sitemap: https://www.4specs.com/sitemap.xml.gz
• Pages with new file names - The error pages from the Google site will let you know what files they think you still have on your site. If there is a new page replacing the old one, set up a 301 server-level redirect to the new page. You can remove this after 3-4 months and replace it with the 410 code discussed below.
• Pages that no longer exist - Google thought 4specs still had a Jobboard and was looking for those pages. I dropped this section 4 years ago. As a result of the sitemap program, I added a server-level code "410 - Gone" that tells the search engine that this entire subdirectory has been removed and to drop the page from the index. You need to do this for all your pages or subdirectories that have been deleted. While a "404 - Page Not Found" would seem to be sufficient, I found many old 4specs pages and subdirectories still in Google's database.
• Delete pages that exist on your server's disk and are no longer used. The AuditMyPC program uses the web to traverse the pages. The Google Python program reads your server's disk and will find files not found from your website. Use the two maps together to identify files to delete from the server. I had to delete or otherwise exclude in the Google sitemap configuration file some old 4specs files I did not want indexed in the sitemap program.
• Exclude images from being indexed - Google will index images and make them searchable. I suggest they be excluded using the robots.txt file and again in the Google sitemap configuration file. There is no benefit to have your images searchable.

Let me know if there are any questions or suggestions. Let me know too when you are successful with this program.

--------------------------------------

Colin Gilboy
Publisher - 4specs
Contact us

9/2008 - Also consider using https://www.xml-sitemaps.com/ to develop sitemaps.