Robots.txt

Root file giving crawlers instructions about which areas of a site they may access.

SFT perspective

A Structure control often confused with indexing control. Blocking crawl is not the same as removing a URL from the index.

Relevant pillars : Structure