A Blanket Block On AI Crawlers
A robots.txt rule meant to stop AI training that also stops the crawlers assistants use to find sources. The site disappears from those answers without anyone noticing.
More customers now ask an assistant which dispensary to try before they ever open a map. The answer names two or three stores and links to a handful of sources. AI search optimization is the work of being one of those names: a site the assistants can read, facts that agree everywhere they look, and pages written so a single paragraph can be quoted. For licensed dispensaries and cannabis brands.
Google, ChatGPT and Perplexity all build answers with large language models, and each one finds pages, pulls out a passage and weighs whether to believe it. Answer engine optimization, or LLM SEO, works on all three steps.
The page has to be indexed by Google, reachable by the assistants' own crawlers, and readable without running scripts, because several AI crawlers do not run JavaScript. That part is technical SEO done properly.
Answers are stitched together from short passages. A page that answers a question in one clear paragraph gets quoted. A page that circles the point for three screens gets skipped.
The assistant checks whether other sources agree: the same name, address and hours on your profile and listings, reviews, and mentions from local press and directories.
Switch between the two versions of a store page. The facts do not change. What changes is whether an assistant can lift one paragraph and trust it enough to name the store.
At Green Harbor we believe cannabis is a journey, and every journey is different. Our passionate team is here for you whatever you are looking for, with an amazing selection that is always changing. Come and see us soon and experience the difference for yourself!
Yes. Green Harbor Eastside stocks low dose gummies and chocolates from 2.5 mg per piece, and staff explain how to start low and wait before taking more. The store is at 120 Main Street and is open until 10 PM.
No clear passage found on the store's own site. The answer cites a directory listing instead, and the store is not named. Source: directory.example
Green Harbor Eastside stocks low dose gummies and chocolates from 2.5 mg per piece, and staff explain how to start low. Source: greenharbor.example/stores/eastside/
None of these are about tricks. They are the same basics that decide ordinary search, applied to a reader that is less patient than Google.
A robots.txt rule meant to stop AI training that also stops the crawlers assistants use to find sources. The site disappears from those answers without anyone noticing.
Hours in an image, the menu in a frame, the address loaded by a script. A crawler that reads only the HTML finds almost nothing. The AI Answer Readability Checker shows what it gets.
Old hours on one listing, a former name on another. When sources disagree, an assistant tends to leave the store out rather than risk a wrong answer.
Welcome copy, slogans and "come see us" with no plain sentence saying what the store has, where it is and when it is open. Rewriting pages to answer first fixes it for search and assistants alike.
Assistants lean on sources that cite each other. A store with reviews but no local press or directory coverage gives them little to corroborate.
An assistant can repeat old hours or a closed location for months. Without regular checks, the first to notice is a customer at a locked door.
A quick reference, and the first thing generative engine optimization checks. Each assistant has its own crawler, its own rules for being left out, and its own place in your reporting. Getting one of these wrong can hide the store from that assistant entirely.
| Answer engine | Reads the site with | Left out by | Shows up in reports as |
|---|---|---|---|
| Google AI Overviews and AI Mode | Googlebot |
noindex or nosnippet on the page | Part of the Web totals in Search Console |
| ChatGPT search | OAI-SearchBot |
A robots.txt block on OAI-SearchBot | Visits referred from chatgpt.com |
| Perplexity | PerplexityBot |
A robots.txt block on PerplexityBot | Visits referred from perplexity.ai |
| Microsoft Copilot | Bingbot |
Anything that keeps the page out of Bing | Visits referred from copilot.microsoft.com |
Crawler names are published by each company and can change. They are checked again at the start of every sprint.
Assistants cross check before they name a store. When the website, the profile and the listings disagree, the store is usually the one left out. Switch between the two states to see what cleanup changes.
| Fact | Website | Business Profile | Cannabis directory | Local news |
|---|---|---|---|---|
| Name | Green Harbor EastsideGreen Harbor Eastside | Green Harbor EastsideGreen Harbor Eastside | Green Harbour DispensaryGreen Harbor Eastside | Green HarborGreen Harbor Eastside |
| Address | 120 Main St120 Main St | 120 Main St120 Main St | 120 Main St120 Main St | 118 Main St120 Main St |
| Friday hours | 9 AM to 10 PM9 AM to 10 PM | 9 AM to 9 PM9 AM to 10 PM | 10 AM to 8 PM9 AM to 10 PM | Not mentioned |
| Phone | (555) 010 4420(555) 010 4420 | (555) 010 4420(555) 010 4420 | (555) 010 3391(555) 010 4420 | Not mentioned |
| Order ahead | YesYes | YesYes | Not listedYes | Not mentioned |
Most of this is ordinary SEO done carefully. The difference is how it is checked: in the assistants themselves, with the questions customers actually ask.
Assistants refresh their sources at different speeds, so changes are measured over months rather than declared done on the day.
Thirty customer questions run in four answer engines, with every mention, citation and wrong fact recorded.
Crawler rules fixed, plain HTML checked, and the store's details made identical on the site and every listing.
Five key pages rewritten to answer first, with checkable facts and compliant wording, and schema validated.
The same questions run again each month, so any change in how the store is named is visible and dated.
AI search is new enough that plenty of reports quote figures nobody can check. These three can be checked, and they are the only ones reported.
Thirty customer questions in four answer engines each month, recording whether the store is named, cited, or missing.
Visits arriving from assistants, reported as their own channel so they are not lost inside referral traffic.
Google counts AI Overviews and AI Mode inside the ordinary Web figures, so they are reported there rather than as a separate number.
The case studies do not include AI search results yet, so none are shown on this page.
A fixed sprint to get the store readable, consistent and quotable, then an optional monthly plan that keeps checking what the assistants say.
AI search work sits on top of ordinary SEO. If Google cannot read the site or the pages aim at nothing, fix that first, and the assistants usually follow.
| What is included | AI search only | Full package |
|---|---|---|
| Prompt tracking in answer engines | Readiness check | |
| Crawler access for AI assistants | ||
| Store details matched everywhere | ||
| Pages rewritten to answer first | Five, then two a month | As the plan needs |
| Technical and rendering audit | ||
| Business Profile and local work | ||
| Starts at | $800 once | $600 a month |
The free check runs a set of customer questions in the main answer engines and tells you where the store is named, where it is missing and which sources they trust instead. You get it in writing, whether or not we work together.
Get Your Free AI Visibility Check Or see every service and the pricing