Website

Connect your website to your AI agent. It crawls your pages and answers from them in your helpdesk, in Slack and on your site, with the page linked.

Website is a knowledge source that turns the pages of your site into answers. Point your agent at your site and it crawls up to 2,000 pages found through your sitemap and links. Pages are usable while the crawl is still running.

Your AI teammate reads the pages of the website you point it at, and answers from them. Every answer links the page it took the answer from.

Your agent answering from crawled pages, with the source page linked.

Connect your website

1

Ask your agent to add your website

Open the chat in your dashboard and ask it to add your website as a knowledge source. It asks which URL to crawl. Prefer to click? Integrations > Website in the left sidebar has the same connect form.

The agent asks for the URL it should crawl.

2

Give the URL

Send the URL you want read. The URL decides the scope of the crawl:

• The crawl starts at the exact URL you give.

• A URL for one section, such as one ending in /docs, holds the crawl to that section.

• The root domain crawls the whole of that domain. A subdomain such as help.yourcompany.com is a separate integration.

Your agent starts the crawl and confirms the website is connected. Pages become searchable while the crawl is still running, so you can keep chatting. The left sidebar now lists the Website integration with your domain.

The crawl runs in the background while the chat carries on.

Check it worked. The integration page shows Connected with your domain, and the Website source shows a document count. From a terminal, eesel status lists the integration as connected with its source counts.

Connect-form options: path filters, custom headers, query strings

The connect form under Integrations > Website has extra controls for narrowing the crawl:

Field	What it does
Include paths	Crawl only URLs whose path matches an entry, for example /help, /docs. Comma-separated. The * and ? wildcards work
Exclude paths	Skip URLs whose path matches, for example /blog, /careers. Exclude wins when both are set. Both empty crawls everything
Custom headers	For sites behind a login: add a Cookie, Basic Authorization or Bearer header and the crawler sends it with every request
Query-string URLs	By default, URLs differing only by query string count as one page and tracking parameters are stripped. Turn this on if query strings select real content, like /resources/?category=manuals

In these fields use paths like /docs, not full URLs. The crawler is already held to the domain you gave.

3

Check what it can read

Integrations > Website shows the connection and its sources. The Website source shows a document count once pages are indexed. Its toggle stops your agent answering from those pages. The two Start a chat buttons open a chat about Website, one for a question and one for setup help.

The Website integration page, with the crawled pages counted under Sources.

4

Tell it how to answer

Tell your agent in chat what to always mention, or always leave out. It saves the rule and applies it from then on.

A standing rule given in chat, saved to the agent's instructions.

The rule then sits on the Instructions page, where you can read or change it.

The Instructions page holds every standing rule your agent follows.

5

Put it to work on your site

Add the eesel chat bubble to your website and visitors can ask there. Your agent answers from the crawled pages, links the pages it used, and follows your instructions. See Chat Bubble for how to add the bubble.

The chat bubble on the site, answering from the pages your agent crawled.

From Claude Code, Cursor or a terminal. The same setup works as commands: eesel login, then eesel integrations connect website, which prints the details it needs. Use eesel integrations download start to start the crawl, eesel status to see the integration and its source counts, and eesel chat to ask it a question. Saving a standing rule has no command, so use the dashboard chat for that step. Full command reference: The eesel CLI. Step-by-step: Set up from Claude Code or the terminal. From a Model Context Protocol (MCP) client: Claude Code and MCP clients.

Sources

Sources are what it learns from:

Source	Description	Requires
Web pages	Pages crawled from the URL you give, found through your sitemap and by following links	The URL, plus custom headers if the site is behind a login

A single crawl indexes up to 2,000 pages. Content stays as it was at the last crawl until you re-crawl. There is no automatic re-crawl, so re-save the integration from Integrations > Website to run a fresh one. A fresh crawl also finds newly linked pages.

One integration covers one domain, so www.yourcompany.com and help.yourcompany.com are two integrations. Giving the same domain again re-crawls into the same integration.

Every other integration on the agent is also a source. So one reply can draw on your website, Confluence, Google Drive, your helpdesk and your uploaded files at once. See Integrations.

You stay in control the whole way. Adjust your AI agent's access to knowledge to your liking.

Limitations

• Up to 2,000 pages per crawl. Pages past the cap are not indexed.

• No automatic re-crawl. Content stays as it was at the last crawl until you re-save the integration.

• Orphan pages are not found. A page has to be linked from a crawled page or listed in your sitemap.

• One domain per integration. A subdomain such as help.yourcompany.com needs its own integration.

• Knowledge only. Website has no triggers and no actions, so nothing on your site starts your agent.

Troubleshooting

Crawler not finding a page?

The page must be linked from another crawled page or listed in the sitemap. Orphan pages are not found. Then check three things:

• The page is not blocked by your include or exclude filters.

• The page does not need a login. If it does, add custom headers.

• The crawl has not hit the 2,000-page cap.

Too many irrelevant pages crawled?

Add exclude paths for the sections you do not want, such as /blog or /careers. Or use include paths to hold the crawl to /help or /docs. Then re-crawl.

Content out of date after a site change?

There is no automatic re-crawl. Re-save the integration from Integrations > Website to run a fresh crawl with the current content.

Seeing duplicate pages that differ only by their URL query?

By default, query-string variants collapse into one page. If you turned on treating query-string URLs as distinct and do not want that, turn it back off and re-crawl.

FAQ

How do I add my website to an AI agent?

Ask your eesel agent in the dashboard chat to add your website, and give it the URL. It starts the crawl and confirms the connection. Prefer to click? Integrations > Website has the same connect form. See Connect your website.

Can I chat with my agent while the website crawl is running?

Yes. Pages become searchable as they index, so your agent can answer from the first pages before the crawl finishes.

Can eesel crawl a website behind a login or a paywall?

Yes. Add custom headers on the connect form: a session Cookie, a Basic Authorization header, or a Bearer token. If the credentials rotate, update the headers and re-save the integration to re-crawl.

Do I need to give eesel my website's sitemap?

No. The crawler picks up your sitemap on its own, and also follows links on the pages it reads.

Can I add more than one website to my agent?

Yes, with one integration per domain. A help center on a subdomain like help.yourcompany.com is its own integration, so give that URL directly.

Why did the website crawl start partway into my site?

The crawl starts at the exact URL you give, so only pages linked from there are found. To crawl the whole site, give the root domain.

Can I use my website knowledge from Claude or ChatGPT?

Yes, through eesel. Once your website is connected, you can talk to your eesel agent from Claude, ChatGPT or Cursor, and it answers from your pages. See Use eesel from Claude and other assistants, Use eesel from ChatGPT and Claude Code and MCP clients.

Related pages

• Helpdesk

• Chat Bubble

• Integrations overview

• Set up from Claude Code or the terminal