Home/HighLevel/Features/Sites/How to train website pages with HighLevel's sitemap powered selection

Funnels, Websites and Forms

How to train website pages with HighLevel's sitemap powered selection

In short

Sitemap powered URL selection lets you preview a website's sitemap, pick exactly the pages you want, monitor how many URLs you can add, and train those pages into a Knowledge Base. You can also fall back to crawling for missing pages and retry any that fail.

AreaSites · Website & Content
Used forSaaS company, Ecommerce store, Consulting firm
Setup steps7 steps in 1 task
Last updatedNovember 17, 2026

01Who this is for

  • Content managers who need to control which site pages feed AI answers, avoiding irrelevant material.
  • Product teams that want to train documentation into a Knowledge Base without exceeding URL quotas.

02Knowledge Base web crawler sitemap powered URL selection, explained

The feature shows a modal with all URLs found in a site's sitemap. Inside the modal you can search, filter, select individual pages or all pages, and see a live counter of how many URLs you have chosen versus how many you are allowed. The selection mode you pick, exact URL, path based, or domain wide, determines which set of pages the crawler starts from.

After confirming your choices, the system trains the selected URLs and displays an operation card that tracks crawling and training status. If the sitemap misses pages, you can launch a crawl that follows internal links to discover additional URLs. Failed pages appear on the card and can be retried without restarting the whole process.

03How businesses use Knowledge Base web crawler sitemap powered URL selection

  • SaaS company. The team opens the Knowledge Base, adds the product site as a website source, selects the domain wide mode, filters the sitemap list to keep only feature pages, and trains them. They watch the operation card to ensure all pages load correctly.
  • Ecommerce store. A manager adds the store’s help center URL, chooses the path based mode for /support, uses the search field to locate specific policy pages, selects them, and trains. They monitor the quota counter to stay under the allowed limit.
  • Consulting firm. The firm adds a client’s landing page with exact URL mode to train a single service page. After training, they test the Knowledge Base retrieval tool to confirm the content appears in AI responses.

04The numbers and rules that matter

  • Up to 4,000 web URLs per Knowledge Base.
  • Sitemap found, 426 pages (example).
  • 0 selected of 3,995 available (example quota counter).

05Before you start

  • A Knowledge Base must exist in the HighLevel account.
  • You need access to AI Agents and the Web Crawler tab.

06How to set up and use Knowledge Base web crawler sitemap powered URL selection in HighLevel

Train website pages using sitemap powered URL selection

  1. Open the Knowledge Base by navigating to AI Agents > Knowledge Base, select the desired Knowledge Base, and open the Web Crawler tab.
  2. Click + Add Website, choose a URL selection mode (Exact URL, All URLs with this Path, or All URLs in this Domain), enter the starting URL, and press Select Pages to Train.
  3. Review the modal that lists sitemap found URLs, noting the total page count displayed at the top.
  4. Use the search bar, adjust rows per page, and navigate with pagination to find relevant URLs; check the boxes for pages you want to include and watch the quota counter update.
  5. If required pages are missing, click Discover by Crawling, let the crawler add new URLs to the list, and refine your selections.
  6. Confirm the selected URLs and click Train Selected; an operation card will appear showing training progress.
  7. When training finishes, examine the results for any failed URLs and use the page level retry action to re train those pages.

07What it looks like in HighLevel

08Settings that matter

  • URL selection mode. Controls whether the crawler uses a single exact page, all pages under a specific path, or every page in the domain as the starting set for discovery.

09Things to know before you rely on it

  • A Knowledge Base can hold up to 4,000 web URLs; the quota counter prevents exceeding this limit (per HighLevel's help center, checked November 2026).
  • Discover by Crawling may take longer than using the sitemap preview because it follows internal links.
  • Only URLs within the selected scope (exact, path, or domain) are considered during crawling.

10If something goes wrong

  • A URL fails during training. Open the Web Crawler tab, locate the failed URL in the page list, and use the retry or refresh action to attempt training again.
  • Desired page is not in the sitemap list. Click Discover by Crawling to let the crawler follow internal links and add missing pages to the selection modal.

11How it connects to the rest of HighLevel

Michael's take

Michael Reimer
Michael ReimerFounder of CRM Pros, HighLevel Commander and Certified Admin

Most people grab the sitemap and hit train without looking at what’s actually there. They end up filling the Knowledge Base with irrelevant pages, waste quota, and then have to prune later.

I would open the Select Pages to Train modal first, filter for the sections that matter, watch the quota counter, and only click Train Selected after I’m sure the list is clean. A quick check of the operation card afterward confirms everything trained as expected.

Get More InformationOpens on HighLevelPublished November 3, 2026 · Updated November 17, 2026

Common questions

Common questions about Knowledge Base web crawler sitemap powered URL selection.

What is sitemap powered URL selection?

It is a Web Crawler workflow that shows URLs found in a sitemap so you can choose which pages to train into a Knowledge Base.

Does this feature change the URL limit per Knowledge Base?

No. The overall per Knowledge Base URL limit stays the same; the modal simply shows how many URLs you have left.

When should I use sitemap preview instead of Discover by Crawling?

Use the preview when the sitemap is complete and covers the sections you need, as it is faster. Use Discover by Crawling when the sitemap is missing pages or the site is highly dynamic.

Can I search the list of pages before training?

Yes. The modal includes a search field that lets you filter URLs by keyword or path.

Is there a way to select all pages at once?

Yes. The modal provides a Select all checkbox; you should verify the quota counter before training all pages.

Next step

Pick the features that matter. Then get the plan that unlocks them.

Explore the features you want, start with the five at the top, and upgrade through my link when you are ready. The bonuses and the team come with it.

The features directory explains every HighLevel feature in plain English. Browse it to see what the platform can do.

Back to the FeaturesSee the full details
⌃

Site preview

Explore the next page

This destination is reserved in the page inventory and is not built yet.