Sitemap and Crawler

For building sitemap (for google), read here.

When you are working with large site (site with lots of pages), break down sitemaps and have sitemap index.

Have robots.txt to tell what to crawl and not.

Have proper Meta Tags for each pages to be properly indexed.

Avoid duplicate contents to be indexed or crawled. Use canonical to let crawler know which URL is the master URL.

Published: 2022-03-26