More information Crawling for searchindex

#How does it work?

#Process

I created a drawing to show what happens when spidering a website to get its content to store it for a website. This is what the ahCrawler does.

Next to this crawling there is a 2nd resource crawling for the link analysis and more.

#Links

#robots.txt

• https://moz.com/learn/seo/robotstxt
• https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt

#x-robots

• https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

#Canonical link

• https://en.wikipedia.org/wiki/Canonical_link_element