InsightsTechnical AEOBy BrandiliteUpdated 9 min read

AI search checklist: can the bots read your website?

Open Brandilite’s AEO service page. It explains who we help, the website work we take on and how to get started. Those details are what we want a search tool to receive.

Now imagine that same address sending the tool to a login screen. The page could still look fine in your browser because you’re already signed in. Rewriting the headline would leave the actual problem sitting there.

Technical page checks: request, access, page identity and readable content

That’s the point of a technical AI search check. Take a page that matters to the business, find out what comes back when it is requested, and follow any problem to the setting or person who can fix it. We’ll use our service page to make the checks less abstract. The faults below are examples to look for, not findings about that page.

Get the right page back before doing anything else

Start with the link you actually give customers. Copy it from the website, rather than guessing the address from a page name in the editor.

For our example, we want the public service explanation. Ending up on the homepage, an old offer or an account screen would be a different result. A redirect isn’t automatically a problem, but where it finishes matters.

Your developer can look at the response code that comes back with the page. A 200 means the request succeeded. You still have to read what succeeded: a response containing “please enable cookies” hasn’t delivered the service explanation you were expecting.

Google’s technical requirements put working, accessible pages and indexable content among the requirements for Search. Meeting those requirements makes a page eligible; Google doesn’t promise to index it.

You don’t need to become the person debugging the server. Send a request someone can answer:

Please check this public page without an account or saved login. Tell me the address where the request ends, the response code and whether the returned content includes our service explanation. If a security rule stops it, show me which rule.

That is more useful than forwarding a screenshot of a bad readiness score.

Keep the returned page with the finding. If the problem only happens for certain requests, the hosting logs may help explain it. A firewall or content delivery service can stop the request before the website application gets a chance to respond. Your developer needs to check that part too.

Leave customer accounts and private material protected. The job here is to make an intended public page available through the intended route.

Check the route before rewriting the answerFive technical checks for a public priority URL: fetch, access, identity, content and release verification.BRANDILITE / AI SEARCH FIELD NOTESCheck the route before rewriting the answer1FetchRight page. Successful response.2AccessIntended search systems allowed.3IdentityCorrect URL and canonical.4ContentUseful answer can be retrieved.5ReleaseFix checked on the live page.
FetchVerify the public response and intended page.
AccessCheck search controls and crawler access.
IdentityResolve indexing and canonical conflicts.
Content & releaseRead the delivered content and verify the final change.

Read what the page says without relying on its design

On our service page, a visitor can read about technical access, page changes, content and measurement. They can follow links to the plans and see the next step. If all a tool received were the logo and a menu, most of that explanation would be missing.

Ask your developer to compare the first HTML response with the page after its scripts have run. In plain language: what arrives straight away, and what has to be added afterwards?

Google can process JavaScript. Its documentation on JavaScript websites describes crawling, rendering and indexing as separate work. That isn’t a reason to tear out every interactive feature. It is a reason to test the important text instead of assuming your browser proves the job is done.

Pick a few details a buyer needs. On a software page, that could be the supported integration, a price rule and a limitation. On a service page, it could be the work included, the information the client needs to supply and what happens after an enquiry.

Find those details in the returned content. If they only appear after opening a tab or playing a video, ask whether a readable explanation can sit on the page as well. Let the video demonstrate the product; keep the basic buying information available to read.

Images need the same thought. A chart can show a change quickly, but a reader still needs the measure, dates and source. Keep the explanation in the page text too, so the information is available when an image cannot be read.

This check sometimes exposes a writing problem as well as a technical one. A page can arrive perfectly and still leave the visitor unsure what you sell. Fixing delivery won’t supply a missing explanation. Our guide to getting seen in AI search covers that part.

Ask which request was blocked

“Allow the AI bots” is a frustratingly loose instruction. Which service are we talking about, and what do we want it to do?

OpenAI documents separate crawlers for search discovery and potential model training. There is also a tool for certain user requests. A company can make different choices about those uses. Check the search crawler when the question is search visibility.

Perplexity has its own crawler documentation, including information for verifying requests. Ask the hosting team to use the current details. A request calling itself a familiar bot name is not enough to establish where it came from.

You want an answer such as, “This rule blocked the search request to the public page.” Now there is something specific to discuss. Which part of the rule needs changing? What should remain protected? How will the developer check the result?

The business owner should decide the intended policy. A developer shouldn’t have to guess whether the company wants its public material available for search, training or both. Put the choice in writing and keep the implementation tied to it.

It’s also worth asking what the audit actually tested. A normal browser request, a tool claiming to be a crawler and a request verified in server logs tell you different things. Keep that distinction in the notes. Otherwise, “it worked when I tried it” can turn into an argument about two tests that were never the same.

Document the access decisionIdentify the request purpose, record the site owner's choice and verify the implementation.BRANDILITE / AI SEARCHDocument the access decision01Name the purposeSearch discovery,training or a userrequest.02Record the owner'schoiceAllowed, restricted orwaiting for review.03Check theimplementationInspect rules and saveevidence of the result.
Name the purposeSearch discovery, training or a user request.
Record the owner’s choiceAllowed, restricted or waiting for review.
Check the implementationInspect rules and save evidence of the result.

Check the quiet settings that survive a redesign

A finished page can carry an instruction that no visitor sees. This is why a design review and a search check need different questions.

Take a page copied from a draft. The words and images are updated, everyone approves the layout, and the old noindex instruction is still attached. That instruction asks a search engine to leave the page out of its index. It can be in the HTML or a response header added elsewhere in the setup.

Ask for both to be checked on the public URL. An editor setting alone won’t show you a rule added by the hosting service. Google’s noindex guidance also explains why the crawler needs access to read the instruction.

The robots.txt file handles access. Blocking a crawler there is not a reliable way to remove an address already discovered in Search. Google explains the difference in its robots documentation. Keep private or unfinished material excluded as intended; find the wrong instruction on the page you actually want discovered.

Then check its canonical URL. That is the preferred version declared for similar content. If you copied a site or moved to a new domain, make sure the new page isn’t still naming the old address. Compare it with the final URL, sitemap entry and links around the site. Google treats these as canonical signals, so it can choose a different version when the signals disagree.

Google has another relevant choice in Search Console. Under Settings, open Search generative AI and check the effective inclusion setting. A site may use the default or inherit a parent property’s choice. An exclusion prevents appearance in the supported AI features, even if ordinary Search still works.

There is one more place copied details can linger: structured data. This is code describing the page. For Brandilite, the company named in it should be Brandilite, with the right address and relevant details. A template that describes a previous business needs correcting even if it passes a syntax test.

Google’s structured data rules require the markup to match the visible content. Leave out unsupported awards or review scores. Its AI search guidance also says a special AI text file or special schema is not required. Ask what an addition represents before paying someone to add it.

Ask for the public URL after the fix

An editor screenshot can show the intended change. You still need to check the page a visitor or tool receives after publication.

Suppose the problem was an old canonical. Open the final page and inspect that value again. If it was a security block, repeat the relevant request. If important text was missing from the first response, compare that response with the expected text. Check the thing that was wrong.

Have a look at another page using the same template too. A fix for one product page should not quietly remove information or change access on the others.

Keep a short record with the original problem, the changed setting, the public URL, the date and the result of the check. Add anything still waiting on a crawler to return. That gives the next person a way to tell what is finished without reopening the whole investigation.

Search Console’s Generative AI performance report can show impressions within its coverage. That is different from confirming a technical fix, and it does not tell you how many enquiries followed. Check the page first; use the appropriate report for the later question you want to answer.

With the page available, go back to the customer. Can they understand the offer, check the important claims and take the next step? Those are the reasons for getting the technical details right. Brandilite’s methodology explains how we connect that work with the rest of an AEO engagement.

Sources and further reading

  1. Google’s technical requirements
  2. documentation on JavaScript websites
  3. documents separate crawlers
  4. crawler documentation
  5. noindex guidance
  6. robots documentation
  7. canonical signals
  8. Search generative AI
  9. structured data rules
  10. AI search guidance
  11. Generative AI performance report

Keep going

Get a free AEO audit.

Put your website into the quick tool. See which public pages it can read and what needs a closer look. You can review the findings before deciding whether you need our help.

Get a free AEO audit