Web ingestion
Crawl a single page, a whole site, or a list of URLs into a Knowledge Base.
Alongside file upload, the Documents tab has an Add URL option for pulling web content directly into a Knowledge Base — a live product page, a documentation site, a batch of known URLs — without downloading and re-uploading anything yourself.
Three modes
- Single URL — crawls exactly one page.
- Root URL — crawls a page and follows links within the same domain, up to a page limit (1–1000, default 10). Options: allow subdomains, ignore links (index the seed page only, don't follow anything).
- URL List — up to 1000 URLs, one per line; each is crawled as its own single-page source.
A submitted URL source appears in the documents table alongside file rows, with a Crawling status badge that updates as pages are discovered and indexed. Root crawls get a View Pages action showing every discovered child page with its own status.
Private and internal addresses are blocked
URLs resolving to a private IP range or localhost are rejected outright — both when you submit
them and again on every re-crawl (in case a domain's DNS changed in the meantime). This isn't
configurable; it's a safety boundary against pointing the crawler at your own internal
infrastructure.
Re-crawling
Re-crawling a source is a clean slate: every chunk previously indexed from that source is deleted before the new crawl begins, then the page(s) are fetched and indexed fresh. There's no incremental/delta mode yet — a re-crawl always processes the full page set again.
A source already crawling can't be re-crawled again until the current run finishes.
Deleting a source
Removes the source and every chunk it produced from the index. Discovered child-page records go with it.