About RankTwigBot
RankTwigBot fetches public web pages when a RankTwig user asks the product to analyze a website or competitor. It is used to build the evidence layer behind crawl findings, page comparisons and content research.
What it does
It follows public HTTP/HTTPS pages within a bounded crawl, records document structure and links, and may inspect sitemap URLs. Competitor crawls use smaller page limits.
Robots and crawl limits
Automated site scans honor robots.txt rules for RankTwigBot or the wildcard user agent, respect supported crawl-delay directives within a safety cap, restrict crawling to the requested origin, and limit page count, depth, response size, redirects and request time.
What it does not do
RankTwigBot does not attempt to bypass authentication, CAPTCHAs, paywalls or private-network restrictions. It does not crawl arbitrary pages continuously; scans are initiated by product users.
User agent
RankTwigBot/0.1 (+<RankTwig origin>/bot)