AI search glossary

    What is robots.txt?

    robots.txt is a file at the root of a website that tells crawlers which parts of the site they may visit.

    Google says robots.txt rules for Googlebot are how site owners control crawling for Search, including its AI features.

    Why it matters for your business

    A single wrong line in robots.txt can block search engines or AI crawlers from your whole site.

    How it works

    1. 1

      User-agent

      Each rule names a crawler, such as Googlebot or OAI-SearchBot.

    2. 2

      Allow and Disallow

      Rules say which paths that crawler may visit.

    Common misunderstandings

    Myth

    robots.txt keeps pages out of search results.

    Reality

    It controls crawling, not indexing. Use noindex to keep a page out of results.

    What to do about it

    • Open yoursite.com/robots.txt and check nothing important is blocked.
    • Decide deliberately which AI crawlers to allow.

    Questions people ask

    Should I block AI crawlers?

    Blocking search crawlers such as OAI-SearchBot can keep you out of AI search answers. Training crawlers are a separate choice.

    See where your business stands

    Run the Free Check

    Sources

    Related terms

    OAI-SearchBotGPTBotnosnippet

    Defined by Dr. Radhakrishnan KG, founder of WebNamaste · Reviewed 6 October 2026