On-site - ROBOTSTXT Factor

A JSON containing basic data about the robots.txt page found on the website. If website does not have a Robots page or the robots is invalid or could not be read, then returns null.



KeyTypeDescription
robotstxtBooleanIndicates if the website contains a Robots.txt page or not.
skippedBooleanIndicates if the robots allows Googlebot rules or not on its Robots.txt page.
validBooleanCheck if the Robots.txt page contains any warnings, if not has, then is valid.
contentObjectAll the disallow content found inside the Robots.txt page.
sitemapObjectA String arraylist that detects all the sitemaps found inside the Robots page.
urlStringThe Robotx.txt URL path to access inside the website.