# Agents

An AI agent can read your site the way a person does, by fetching its pages. Sitepaste also publishes the Markdown behind your pages and a list of them, so an agent can read your content directly when it prefers to.

## Each page's Markdown

Every page has its Markdown source at `index.md` under its own address. The page at `https://example.com/docs/install/` has its source at `https://example.com/docs/install/index.md`.

The source is published whether or not the page shows a Copy Markdown button. That setting only decides whether visitors see the button. The source is published as you wrote it, so an HTML comment such as `<!-- note -->`, which the page itself never shows, can be read in its `index.md`. Keep private notes out of a page's content.

Product pages have no `index.md`, because they are built from their settings rather than from Markdown.

## Following the site in Markdown

The home page, each section, and each tag listing has an `index.md` too, which links every page under it: by that page's own `index.md`, or by its web address for a product page, which has none. The home page's `/index.md` holds the home page's content, then links each section and top-level page, and the site's `llms.txt`. An agent can start at `/index.md` and reach every page without reading any HTML.

## llms.txt

The site publishes `/llms.txt`, which follows the format described at [llmstxt.org](https://llmstxt.org/). It holds the site's name and description, then the site's pages, grouped by section, each linked to its `index.md`. An agent can read this one file to see what the site covers and then fetch only the pages it needs.

`llms.txt` lists each page that has an `index.md` and that search engines may index. It does not list the section and tag listings, which an agent reaches through `/index.md`. On the Free plan, where pages are not indexed by search engines, it lists no pages.

## Protected content

A protected page's `index.md` is protected in the same way as the page itself. On a password-protected or workspace-protected site, every `index.md` is protected as well, and so is `llms.txt`. Once the visitor is signed in or has entered the password, they list the protected pages too. On a public site, the listings and `llms.txt` leave out any page that has its own password. See [Protection](/concepts/protection).

## Bots in your metrics

On the Pro plan, the Metrics page counts bots apart from your visitors. Its Bots card lists each bot by name with its number of hits, in two groups:

- AI and agents: assistants such as ChatGPT or Claude fetching a page to answer a question, coding agents, and the crawlers that train AI models or gather pages for AI search answers.
- Search engines and other bots: crawlers that index pages for search, link previews, and any other bot.

The card also lists the pages agents read most. The info icon next to a bot's name shows who runs it, what kind of bot it is, and how many of its requests asked for Markdown. The same breakdown is available through the [metrics API](/api/data/get-metrics).