> For the complete documentation index, see [llms.txt](https://docs.eesel.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eesel.ai/integrations/knowledge/website.md).

# Website

Point your agent at your website and it reads the pages and answers from them, with a link to the page each answer came from.

Your AI teammate reads the pages of the website you point it at, and answers from them. Every answer links the page it took the answer from.

Nothing runs on its own. Website is knowledge only: no triggers and no actions.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-9855977a066eb01e13157560f804f812997e5ba1%2Fwebsite-hero.png?alt=media" alt="A Website knowledge source card listing crawled pages, feeding an agent that answers a question in Slack and cites the page it used"><figcaption><p>Your agent answering from crawled pages, with the source page linked.</p></figcaption></figure>

## Quick start

{% stepper %}
{% step %}

### Ask your agent to add your website

Open the chat in your dashboard and ask it to add your website as a knowledge source. It asks which URL to crawl. Prefer to click? **Integrations > Website** in the left sidebar has the same connect form.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-e22d3abed61a8ab6d944c841ceebff470faedf30%2Fwebsite-connect-chat.png?alt=media" alt="The dashboard chat where the user asks to add their website as a knowledge source and the agent asks which URL to crawl"><figcaption><p>The agent asks for the URL it should crawl.</p></figcaption></figure>
{% endstep %}

{% step %}

### Give the URL

Send the URL you want read. The URL decides the scope of the crawl:

* The crawl starts at the exact URL you give.
* A URL for one section, such as one ending in `/docs`, holds the crawl to that section.
* The root domain crawls the whole of that domain. A subdomain such as `help.yourcompany.com` is a separate integration.

Your agent starts the crawl and confirms the website is connected. Pages become searchable while the crawl is still running, so you can keep chatting. The left sidebar now lists the Website integration with your domain.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-05acce590fc684ce4ba55606d661f11bb076e2ed%2Fwebsite-crawl-started.png?alt=media" alt="The dashboard chat where the agent reports the crawl is running, confirms Website connected, and suggests test questions, with the Website integration in the sidebar"><figcaption><p>The crawl runs in the background while the chat carries on.</p></figcaption></figure>

**Check it worked.** The integration page shows **Connected** with your domain, and the Website source shows a document count. From a terminal, `eesel status` lists the integration as connected with its source counts.

<details>

<summary>Connect-form options: path filters, custom headers, query strings</summary>

The connect form under **Integrations > Website** has extra controls for narrowing the crawl:

| Field                 | What it does                                                                                                                                                                                   |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Include paths**     | Crawl only URLs whose path matches an entry, for example `/help, /docs`. Comma-separated. The `*` and `?` wildcards work                                                                       |
| **Exclude paths**     | Skip URLs whose path matches, for example `/blog, /careers`. Exclude wins when both are set. Both empty crawls everything                                                                      |
| **Custom headers**    | For sites behind a login: add a `Cookie`, Basic `Authorization` or `Bearer` header and the crawler sends it with every request                                                                 |
| **Query-string URLs** | By default, URLs differing only by query string count as one page and tracking parameters are stripped. Turn this on if query strings select real content, like `/resources/?category=manuals` |

In these fields use paths like `/docs`, not full URLs. The crawler is already held to the domain you gave.

</details>
{% endstep %}

{% step %}

### Check what it can read

**Integrations > Website** shows the connection and its sources. The Website source shows a document count once pages are indexed. Its toggle stops your agent answering from those pages. The two **Start a chat** buttons open a chat about Website, one for a question and one for setup help.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-f2dbac2f7bf7fdfb6869deaf4c2b82dcccc9672e%2Fwebsite-integration-page.png?alt=media" alt="The Website integration page showing Connected with the domain, two Start a chat buttons, and a Sources list with a Website row, its document count and a toggle"><figcaption><p>The Website integration page, with the crawled pages counted under Sources.</p></figcaption></figure>
{% endstep %}

{% step %}

### Tell it how to answer

Tell your agent in chat what to always mention, or always leave out. It saves the rule and applies it from then on.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-5c0e95f256c95061970a020ae6b3de2699995ec8%2Fwebsite-instruction-chat.png?alt=media" alt="The dashboard chat where the user gives a standing rule about what to mention in answers, and the agent confirms it saved the rule to its instructions"><figcaption><p>A standing rule given in chat, saved to the agent's instructions.</p></figcaption></figure>

The rule then sits on the **Instructions** page, where you can read or change it.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-bd5bb320b70878c986cebb1954757feb4d7fb49b%2Fwebsite-instruction-saved.png?alt=media" alt="The Instructions page showing the saved standing rule as editable text, marked Saved"><figcaption><p>The Instructions page holds every standing rule your agent follows.</p></figcaption></figure>
{% endstep %}

{% step %}

### Put it to work on your site

Add the eesel chat bubble to your website and visitors can ask there. Your agent answers from the crawled pages, links the pages it used, and follows your instructions. See [Chat Bubble](/integrations/chat-surfaces/chat-bubble.md) for how to add the bubble.

<figure><img src="https://3732419023-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2xXO0947TYoPIhGoBgSE%2Fuploads%2Fgit-blob-daec696d4918af074672745648148fa787b0a7c9%2Fwebsite-chat-bubble.png?alt=media" alt="A website with the eesel chat bubble open, where a visitor has asked a question and the agent has answered with links to the crawled pages"><figcaption><p>The chat bubble on the site, answering from the pages your agent crawled.</p></figcaption></figure>
{% endstep %}
{% endstepper %}

{% hint style="info" %}
**From Claude Code, Cursor or a terminal.** The same setup works as commands: `eesel login`, then `eesel integrations connect website`, which prints the details it needs. Use `eesel integrations download start` to start the crawl, `eesel status` to see the integration and its source counts, and `eesel chat` to ask it a question. Saving a standing rule has no command, so use the dashboard chat for that step. Full command reference: [The eesel CLI](/apis-and-developer-resources/cli.md). Step-by-step: [Set up from Claude Code or the terminal](/getting-started/set-up-from-a-coding-agent.md). From a Model Context Protocol (MCP) client: [Claude Code and MCP clients](/apis-and-developer-resources/claude-code-and-mcp-clients.md).
{% endhint %}

## Sources

Sources are what it learns from:

| Source        | Description                                                                            | Requires                                                   |
| ------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| **Web pages** | Pages crawled from the URL you give, found through your sitemap and by following links | The URL, plus custom headers if the site is behind a login |

A single crawl indexes up to **2,000 pages**. Content stays as it was at the last crawl until you re-crawl. There is no automatic re-crawl, so re-save the integration from **Integrations > Website** to run a fresh one. A fresh crawl also finds newly linked pages.

One integration covers one domain, so `www.yourcompany.com` and `help.yourcompany.com` are two integrations. Giving the same domain again re-crawls into the same integration.

Every other integration on the agent is also a source. So one reply can draw on your website, Confluence, Google Drive, your helpdesk and your [uploaded files](/integrations/knowledge/uploading-files.md) at once. See [Integrations](/integrations/overview.md).

## Troubleshooting

<details>

<summary>Crawler not finding a page?</summary>

The page must be linked from another crawled page or listed in the sitemap. Orphan pages are not found. Then check three things:

* The page is not blocked by your include or exclude filters.
* The page does not need a login. If it does, add custom headers.
* The crawl has not hit the 2,000-page cap.

</details>

<details>

<summary>Too many irrelevant pages crawled?</summary>

Add exclude paths for the sections you do not want, such as `/blog` or `/careers`. Or use include paths to hold the crawl to `/help` or `/docs`. Then re-crawl.

</details>

<details>

<summary>Content out of date after a site change?</summary>

There is no automatic re-crawl. Re-save the integration from **Integrations > Website** to run a fresh crawl with the current content.

</details>

<details>

<summary>Seeing duplicate pages that differ only by their URL query?</summary>

By default, query-string variants collapse into one page. If you turned on treating query-string URLs as distinct and do not want that, turn it back off and re-crawl.

</details>

## FAQ

<details>

<summary>Can it crawl a site behind a login or a paywall?</summary>

Yes. Add custom headers on the connect form: a session `Cookie`, a Basic `Authorization` header, or a `Bearer` token. If the credentials rotate, update the headers and re-save the integration to re-crawl.

</details>

<details>

<summary>Do I need to give it my sitemap?</summary>

No. The crawler picks up your sitemap on its own, and also follows links on the pages it reads.

</details>

<details>

<summary>Can I crawl more than one website?</summary>

Yes, one integration per domain. A help center on a subdomain like `help.yourcompany.com` is its own integration, so give that URL directly.

</details>

<details>

<summary>Why did the crawler start partway into my site?</summary>

The crawl starts at the exact URL you give, so only pages linked from there are found. To crawl the whole site, give the root domain.

</details>

<details>

<summary>Can I still chat while the crawl is running?</summary>

Yes. Pages become searchable as they index, so your agent can answer from the first pages before the crawl finishes.

</details>

<details>

<summary>How is this billed?</summary>

Nothing extra for the knowledge source. Billing is pay-as-you-go: a ticket your agent handles is one task, however many replies happen on it. Chats you run yourself from the eesel dashboard are free. See [Pricing](/pricing/overview.md) for rates, the annual commitment, and Enterprise.

</details>

## Related pages

* [Helpdesk](/use-cases/helpdesk.md)
* [Chat Bubble](/integrations/chat-surfaces/chat-bubble.md)
* [Integrations overview](/integrations/overview.md)
* [Set up from Claude Code or the terminal](/getting-started/set-up-from-a-coding-agent.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.eesel.ai/integrations/knowledge/website.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.