Enter your email address:

Thursday, March 26, 2009

Making A 100% Crawlable Site

Many webmasters believed that great content is the key to Search Engine Optimization but they have forgotten an even more important factor, how easy can the bots crawl their sites? Making a 100% crawlable site should be the top priority for all the webmasters. There is no point having a unique and fresh content but cannot be searched from the search engine index.

In order to ensure that bots can crawl a site successfully, one must make sure that all pages can be found by going from pages to pages via the inter linking mechanism. It is recommended to use text links as a form of linking between all the internal pages. Webmasters can also provide a sitemap that lists down all the pages’ links in a page to facilitate this indexing. One point to take note is that if the site has more than 100 links, it is advisable to split the sitemap into several pages with each page containing no more than 100 links. A page with more than 100 links may be classified as ‘Links Farm’ by the search engine. This is stated officially in the Google Webmaster Guidelines.

Google Sitemap is another great tool for them to index a new site. Creating a Google sitemap is encouraged because it tells the search engine what are the pages in a site and how often are the content updated. This is particularly useful when some pages are not linked to within the site.
Many sites have faced crawling problems because of the way they linked up the internal pages. Search Engine bots have difficulties indexing JavaScript and Flash navigation menu. If it is not possible to remove this kind of navigation in a site, it is advised to implement a text-link navigation system in the footer of a site. This will help the bots to index the site easily.

According to Google Webmaster Guidelines, it is advised to have a static and short HTML link destination rather than a dynamic URL. Dynamic URL especially those with tagged session identifiers will not be indexed because bots will ignore these pages. It is also encouraged to avoid using the ‘&id=’ parameter when passing variables between pages as Google does not include them into the index. If parameters are needed to pass between pages, meaningful parameter like ‘&count=’ can be used instead.

When the bots conduct a visit, they will crawl the page like what a general user will see in their browser. Therefore, the bots will never index a password-protected page. If the objective is to get the site indexed by the search engine, it is advisable to remove any password-protected pages to allow access for the bots.

A site should always be search engine friendly so that users can locate the content easily. To conclude, search engine likes to index simple and content rich pages. A simple and crawlable site with great content is the essential criteria to Search Engine Optimization.


Niema said...

Hi i have a problem with this crawl, I make 2 blog for 2 months. But my blog http://winnie-manage.blogspot.com havent GIP yet. I dont know what to do. I tried to put it in google analytics but cant help. Can you give me any suggestion. My other new blog has GIP 16 why another still 0. Thanks a lot

Kris said...

ok Niema, just see my new posts

Anonymous said...

hi kris,
i have also same problem with crawl my pages, i made one blog but my pages not index in google and yahoo search engines, my blog name is http://voip-ccvp.blogspot.com and i have not recieved amount of web traffic or users. pls help me what i will do for drive traffic on my blog.

Reetha said...

Dear blogger, I suggest you to do two things quickly to get much traffic.
1. Go to this website, and then join and install their widget. Dont forget to read free ebook provided by them, on how to improve traffic. Everything is free. http://tinyurl.com/rushtraffic

2. Just read this and Do some simple steps. This will help to get indexed quickly. http://businessvartha.blogspot.com/2009/08/one-step-free-submission-of-blog-or.html

3. Please add me in businessvartha@yahoo.com ( I have something more to say )

Blog Widget by LinkWithin