On-site - CRAWLABLE Factor

A JSON containing data about all the robots technology found on the given website page. Please note that each robot can be configured differently, like on the example below which is configured to crawl all the page. If any robots crawler was found or the website content could not be load, then returns null.



KeyTypeDescription
robotsObjectGooglebot crawler found on the website page URL and how it is configured.
googlebotObjectGooglebot crawler found on the website page URL and how it is configured.
googlebot_newsObjectGooglebot News crawler found on the website page URL and how it is configured.
googlebot_imagesObjectGooglebot Image crawler found on the website page URL and how it is configured.
slurpObjectSlurp crawler found on the website page URL and how it is configured.
msn_botObjectMSN bot crawler found on the website page URL and how it is configured.
teomaObjectTeoma crawler found on the website page URL and how it is configured.
robots_headerObjectRobots Header crawler found on the website page URL and how it is configured.