Robots.txt Tester & Validator
Processed in your browser
Test and validate robots.txt crawl rules against specific User-Agents and URL paths.
BLOCKED (Disallowed)
Matched rule: Disallow: /admin/ for User-agent: *
⚡
Direct In-Browser Handoff
DevOps & System Utility Pipeline
Chain this tool with complementary private browser utilities for a complete zero-server workflow:
Helpful Examples
Standard Site Robots.txt
Block admin and allow public paths
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/How to Use the Robots.txt Tester & Validator
Parse robots.txt files in your browser. Test whether Googlebot, Bingbot, or custom web crawlers are allowed or disallowed on specific URLs with step-by-step directive matching.
Privacy & Data Processing Notice
No user data leaves your browser. Computation is executed entirely on your device. We do not store, view, or transmit your inputs over our network. Your computation remains on your personal machine.
Frequently Asked Questions
How does the robots.txt tester evaluate rules?
It matches the specified User-Agent and path against Disallow and Allow directives in accordance with the Google RFC 9309 robots exclusion specification.
Are my robots.txt rules sent to any server?
No. All parsing and URL matching run 100% inside your browser under our P0 privacy standard.
Related Utilities
Frequently paired browser tools in Developer Tools