Loading...
Discoverability surfaces
A plaintext file at the root of a website that tells web crawlers which paths they may or may not fetch. The format is RFC 9309. VC Deal Flow Signal's robots.txt explicitly allow-lists 30+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, GoogleOther, Mistral, Cohere, Apple-AI, Meta-ExternalFetcher and others) by name rather than relying on the wildcard, and disallows authentication, webhook, cron, and dashboard surfaces.
Programmatic SEO, AEO, GEO, AIO, and the schemas behind them.
A content strategy that generates hundreds or thousands of search-optimized pages from structured data using templates.
The practice of structuring website content so that AI assistants and large language models (LLMs) can accurately cite it when answering user questions.
An open protocol that allows websites to notify search engines (Bing, Yandex, Seznam, Naver, and others) about new or updated content in real time.
Structuring content so that answer engines — Google's People-Also-Ask, Reddit pull-quotes, Quora top answers, ChatGPT search results, Perplexity citations — can extract a complete, self-contained answer in 40–80 words.
The subset of GEO/AEO targeted specifically at Google's AI Overviews (formerly SGE).
A Schema.
JavaScript Object Notation for Linked Data — the W3C-standard syntax for embedding structured data in web pages.
A Schema.
This definition is published under CC BY 4.0. Cite as:
The Data Nerd. "robots.txt." VC Deal Flow Signal Glossary, https://signals.gitdealflow.com/define/robots-txt.
The free Acceleration Watch turns terms like robots.txt into five named, accelerating startups every Sunday — translated into plain English, 21 to 47 days before the deck circulates. No code-reading, no card.