Agents
An AI agent can read your site the way a person does, by fetching its pages. Sitepaste also publishes the Markdown behind your pages and a list of them, so an agent can read your content directly when it prefers to.
Each page’s Markdown
Every page has its Markdown source at index.md under its own address. The page at https://example.com/docs/install/ has its source at https://example.com/docs/install/index.md.
The source is published whether or not the page shows a Copy Markdown button. That setting only decides whether visitors see the button. The source is published as you wrote it, so an HTML comment such as <!-- note -->, which the page itself never shows, can be read in its index.md. Keep private notes out of a page’s content.
Product pages have no index.md, because they are built from their settings rather than from Markdown.
Following the site in Markdown
The home page, each section, and each tag listing has an index.md too, which links every page under it: by that page’s own index.md, or by its web address for a product page, which has none. The home page’s /index.md holds the home page’s content, then links each section and top-level page, and the site’s llms.txt. An agent can start at /index.md and reach every page without reading any HTML.
llms.txt
The site publishes /llms.txt, which follows the format described at llmstxt.org. It holds the site’s name and description, then the site’s pages, grouped by section, each linked to its index.md. An agent can read this one file to see what the site covers and then fetch only the pages it needs.
llms.txt lists each page that has an index.md and that search engines may index. It does not list the section and tag listings, which an agent reaches through /index.md. On the Free plan, where pages are not indexed by search engines, it lists no pages.
Protected content
A protected page’s index.md is protected in the same way as the page itself. On a password-protected or workspace-protected site, every index.md is protected as well, and so is llms.txt. Once the visitor is signed in or has entered the password, they list the protected pages too. On a public site, the listings and llms.txt leave out any page that has its own password. See Protection.
Bots in your metrics
On the Pro plan, the Metrics page counts bots apart from your visitors. Its Bots card lists each bot by name with its number of hits, in two groups:
- AI and agents: assistants such as ChatGPT or Claude fetching a page to answer a question, coding agents, and the crawlers that train AI models or gather pages for AI search answers.
- Search engines and other bots: crawlers that index pages for search, link previews, and any other bot.
The card also lists the pages agents read most. The info icon next to a bot’s name shows who runs it, what kind of bot it is, and how many of its requests asked for Markdown. The same breakdown is available through the metrics API.