About our scanner
HLAIAuditBot
HLAIAuditBot is the scanner behind the H/L AI Readiness Audit. It runs when someone asks this site to audit a website, and checks whether AI crawlers and AI agents can reach, read and use that site.
What it does
- It reads robots.txt, the sitemaps and every page it can find by following links, up to 50,000 pages a scan.
- It opens a sample of pages in a headless Chrome browser, which identifies itself with HLAIAuditBot added to a standard Chrome user agent.
- It sends a few requests with the user agents of AI bots, such as OAI-SearchBot and PerplexityBot, to see how the site answers them. The report labels each one as an unverified client using that user agent, because those requests come from us, not from the AI company.
- It never submits forms, starts chats, adds to a cart or creates accounts. Its browser stops at the submit button.
User agent
Plain requests use this user agent:
Mozilla/5.0 (compatible; HLAIAuditBot/1.0; +https://hl-tda.com/bot)
When a signing key is configured, those requests also carry a Web Bot Auth signature for this site. The public key is at /.well-known/http-message-signatures-directory. Requests that use another bot's name are not signed.
How fast it crawls
It starts at one request a second per host, with at most two requests in flight. While a site answers quickly and without errors, the crawl speeds up in steps to at most 4 requests a second, 4 in flight, and it drops back a step after any server error or slow response. A 429 or 503 sends it back to one request a second for the rest of the scan, and it honors Retry-After. If robots.txt sets a Crawl-delay, it never speeds up and waits that long instead. It stops after repeated server errors.
How it uses robots.txt
The crawl follows the robots.txt rules for HLAIAuditBot, or the rules for every bot (*) when there's no HLAIAuditBot group. The homepage, robots.txt, the sitemaps and up to 10 key pages are fetched as user-triggered requests, because a person asked for the scan.
How to block it
Add this to your robots.txt file:
User-agent: HLAIAuditBot Disallow: /
A group that names HLAIAuditBot stops the crawl and the key-page requests. When someone asks for a scan, the scanner still requests the homepage, robots.txt, the sitemaps and a few discovery files such as llms.txt, and sends the AI bot user-agent requests to the homepage and robots.txt, so the report can show what the site serves.
Where requests come from
This scanner's requests come from 136.117.38.23.
Contact
Questions or problems with HLAIAuditBot: NewBusiness@HL.Agency.