Sitemap and Crawler

For building sitemap (for google), read here.

https://developers.google.com/search/docs/advanced/sitemaps/overview?hl=en

When you are working with large site (site with lots of pages), break down sitemaps and have sitemap index.

https://developers.google.com/search/docs/advanced/sitemaps/large-sitemaps?hl=en

Have robots.txt to tell what to crawl and not.

https://developers.google.com/search/docs/advanced/robots/intro?hl=en

Have proper Meta Tags for each pages to be properly indexed.

https://developers.google.com/search/docs/advanced/crawling/special-tags?hl=en

Avoid duplicate contents to be indexed or crawled. Use canonical to let crawler know which URL is the master URL.

https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls?hl=en

Published: 2022-03-26