Robots.txt can be a file that exists on the root Listing of every Web page and can be utilized to instruct serps on which directories/files of the web site they can crawl and involve inside their index. Search engines like yahoo have only one purpose only. They intention to supply https://news.rafeeg.ae