Robots.txt is often a file that exists on the foundation directory of every website and can be used to instruct search engines like yahoo on which directories/files of the web site they are able to crawl and include things like within their index. A significant explanation that SEO matters is https://news.rafeeg.ae