You may add these web pages to the file to get explicitly disregarded. Robots.txt documents use anything known as the Robots Exclusion Protocol. This website will conveniently generate the file to suit your needs with inputs of internet pages for being excluded. In case you’re looking to exclude content from https://rankingseotoolz.com/robots-txt-generator