Can ChatGPT find your public content?
Hong Kong businesses need to separate OAI-SearchBot, GPTBot and noindex.

Ask three questions for every public page

Before changing robots rules, record these three intentions for each URL:

This “page intent × crawler purpose × exposure signal” table is CLEARCRAFT’s internal organising framework. It is not an official score or guarantee from any platform.

OAI-SearchBot: can the content be discovered for search?

OpenAI’s Publisher FAQ says that sites should not block OAI-SearchBot if they want content to be available for summaries and snippets in ChatGPT. This is one condition for public content to be read; it is not a promise of indexing, summarisation, citation or traffic.

A Hong Kong beauty, dental, or engineering company that wants its public service descriptions, location information and verifiable facts to participate in search should therefore treat “may OAI-SearchBot read this?” as a page-level policy. It should not block every AI crawler without first deciding the intent of each page.

GPTBot: is potential training allowed?

OpenAI assigns GPTBot to a different path: a site can use it to express that content should not be used for potential training. That is not the same purpose as search through OAI-SearchBot.

The practical approach is not to default to “allow everything” or “block everything.” A content owner should first confirm which pages contain public service information, which are for clients or internal use only, and which require separate handling. If a page is not suitable for public access, the sound starting point is usually not to publish it as an indexable page, rather than expecting a crawler rule to replace access control.

noindex: when even the title and link should not appear in ChatGPT Atlas

OpenAI also notes specifically for ChatGPT Atlas that, even when a crawler is blocked, Atlas may learn about a URL through a third-party search provider or another page and may show only its title and link in a relevant result. Its documentation recommends noindex for a page that should not appear in this way in Atlas. The crawler must still be able to read the page to see that signal.

This creates an important practical check. If a company says both “do not show this page in Atlas results” and “do not let any crawler read this page,” the settings may conflict. First decide whether the page should be public at all. Client material, authenticated content, draft quotations and internal documents should be protected by suitable access controls rather than treated as ordinary public pages. This Atlas-specific guidance should not be treated as a claim about every platform.

Bilingual Hong Kong sites need aligned policies too

Hong Kong websites often publish Traditional Chinese and English pages. Google’s multilingual-site guidance recommends separate URLs for language versions, suitable hreflang annotations and visible content in the stated language. It also advises against automatically redirecting visitors based on an assumed language, because users and crawlers may be unable to reach every version.

For public-content policy, the implication is simple. If a Traditional Chinese service page is searchable but the English counterpart accidentally carries noindex, or if the two versions point to different facts, a platform cannot be expected to repair the mismatch. Check the same items for every language pair: public intent, canonical, hreflang, visible body content, robots or page directives, and internal links.

Minimum pre-publication checklist

1. Mark the intended status of every important URL for public search, training, agent access and private content.
2. Review OAI-SearchBot, GPTBot and other crawler rules separately; do not make one blanket “AI bot” decision.
3. Check for contradictions among robots.txt, page meta or HTTP X-Robots-Tag, access controls, and WAF or bot rules.
4. Compare the public intent and technical annotations of each Traditional Chinese and English page pair.
5. Record crawler requests, AI referrals, links in answers, and actual enquiries or sales as separate measurements.

These checks reduce contradictory settings but do not replace a platform’s display decision. Platforms decide answers according to the query, time, market, language, available sources and other signals. A website can make its public information and restrictions clear; it cannot dictate the result.

Next step

If you are organising public service pages, start with a “URL — public intent — search crawler — training crawler — page directive — owner” register and check whether every field agrees. For the division among Search, Agent and Training crawlers, read AI crawlers are not all the same: how to manage Search, Agent and Training separately. For source-review workflow, see How should Hong Kong businesses verify the sources in AI-search answers?.