Adding Content
Crawling, typed notes and file uploads — three ways to feed the agent.
Your agent is instructed to answer only from content you've connected and not to make things up — a question your content doesn't cover is declined with an offer to reach your team. There are three ways to feed it, and it uses them together when replying to visitors:
- Website crawling — the agent reads your public website automatically.
- Notes you type — a few lines for the things your site and files don't say, in Dashboard → Knowledge Base.
- File upload — you add documents (PDFs, manuals, policies) that aren't on your site, in the same place.
Most of this is automatic: connect a website and the first two thirds of the work happen without you. Typing is there for the gaps, not as the main way in.
This page explains how each works, how content stays fresh, and the per-plan limits that apply.
How your content becomes answers
Crawled pages, typed notes and uploaded files are indexed together and searched as one pool. They are not, however, weighted equally. When two passages are about equally relevant, a typed note is preferred over a crawled page, and a crawled page over an uploaded file. It is a thumb on the scale rather than a veto: relevance still does most of the work, so a passage that matches clearly better still wins.
Because the agent searches by meaning as well as by wording, your content doesn't have to phrase things the way visitors do — but the underlying facts do have to exist in your crawled pages, uploaded files or typed notes. A custom welcome message or suggested question about a topic your content doesn't cover gives the agent no facts to answer with.
For the rest of the story — how a passage is found and ranked, the Sources control a visitor can open under a reply, what happens when your content doesn't cover the question, and when a browsing question returns visual cards instead of text — see How Answers Work.
Website crawling
When you add a website and activate your plan, NovixAgent AI crawls it and indexes the pages it can reach, up to your plan's page and depth limits (see How much of your site is crawled) — no configuration required. This is the first thing that happens after payment, and you watch it live on the website card in Dashboard → Workspaces: Reading your content → Indexing knowledge → Finalizing assistant → Knowledge base ready. The agent can already answer while indexing finishes.
What gets crawled
- Public pages reachable from your home page, discovered via your sitemap first (falling back to following internal links).
- HTML content — headings, paragraphs, lists, and tables (pricing tables, comparison grids — these are extracted specially so plan/price pairs survive).
- JavaScript-rendered pages — the crawler uses a real headless browser and automatically detects when a page needs full rendering (React/Vue storefronts, Shopify JS themes), so modern single-page sites are read correctly.
- Documents linked from your pages — PDF, DOCX, and TXT files linked on your site are auto-downloaded (up to 25 MB each), their text extracted, and indexed alongside the page content. No setup needed.
The crawler also automatically classifies your business type (e-commerce, restaurant, healthcare, real estate, and so on) and tunes how it reads your site accordingly — which pages to skip (carts, checkouts, admin areas), how deep to go, and whether to fully render pages.
What does not get crawled
- Pages behind a login, or that require cookies/session to view — public pages only.
- Admin, cart, checkout, sign-in, and other transactional or account paths — these are excluded on purpose (along with common tracking-parameter and feed URLs) so the agent learns your content, not your plumbing.
- Linked documents larger than 25 MB (large legal PDFs may exceed this — Storage and file size says what to do with a document that size).
- Images and embedded media — there's no OCR, so text that lives only inside an image won't be read. Upload a text version if you need it.
How much of your site is crawled
How much of a large site the crawler covers in a single pass scales with your plan (see the per-plan limits below for the numbers). Depth is how many link-hops from the home page the crawler will follow; pages is the hard ceiling on how many it indexes. Most small and mid-size sites fit comfortably inside the smallest plan's limits. If your site is very large and you're hitting the page cap, a higher plan crawls more of it — and upgrading starts a fresh crawl to the new limit, unless your last crawl already read at least that many pages.
Keeping crawled content fresh — automatic re-crawling
As long as your site stays reachable, re-crawls happen without you. NovixAgent AI keeps your knowledge current in two ways.
Scheduled re-crawls. On a recurring schedule set by your plan (see the per-plan limits below), the crawler revisits your site and re-indexes what changed. Higher plans keep the agent's knowledge fresher. This is the fastest schedule your plan allows; the crawler only actually re-indexes when something changed — see below.
Smart change detection. Before every scheduled crawl, NovixAgent AI runs a lightweight check to see whether anything is actually different — comparing your sitemap, HTTP headers, and a content hash of your pages. If nothing changed, it skips the re-index entirely, which saves your crawl budget and processing cost. When a page has changed, only the changed pages are re-embedded; unchanged pages are left untouched.
If we repeatedly can't reach your site, automatic crawling pauses. After three consecutive failed checks — the site unreachable, erroring, or refusing the crawler — NovixAgent AI stops scheduling re-crawls for it rather than retrying a dead endpoint indefinitely. This is a failure state, not a saving: a paused site is no longer re-checked, so it does not un-pause itself when the problem is fixed. Your existing answers keep working, but new and changed pages stop being picked up. If your content has stopped updating and the site is healthy again, email support@novixagent.ai with your Website ID and we'll resume it.
If a scheduled update fails, a retry happens automatically unless crawling has paused as described above — the website card shows "Knowledge update failed", and your previously-indexed content keeps serving answers in the meantime, so visitors keep getting answers.
Notes you type
Some things aren't written down where the agent can read them: the price you quote over the phone, the policy you explain in person, the hours that differ from the ones on your site. Dashboard → Knowledge Base has a short text field — "Anything we missed" — for that.
It's optional, and it isn't meant to replace your website: the agent already learns on its own from what you connect. Type a few lines only where there's a gap.
When it matters most
- You have no website yet. A workspace connected only through WhatsApp Business has no website for the crawler to read. Without a few typed lines or an uploaded document, the agent has no content to answer from: it can take a message and call a human. This field is the fastest way to make it useful. The same applies to a workspace connected only through Instagram DMs or Facebook Messenger.
- Your site is out of date. If a page still shows last season's price and your note says the new one, the note is the one the agent should reach for — typed knowledge carries the heaviest weight of the three sources. It tips a close call rather than overriding the page outright, so where the two genuinely conflict it is still worth correcting the page as well. Correcting one line here is faster than a site edit and a re-crawl.
- The answer lives in your head. Delivery radius, what you will and won't accommodate, the notice you need for a custom order.
How to use it
- Go to Dashboard → Knowledge Base and pick the website in the workspace switcher at the top of the left-hand navigation.
- Type into "Anything we missed". Plain sentences work best — no formatting, no structure required. A short paragraph is enough to start.
- Click Save and learn. The note appears in the list below as Your notes and moves from Learning... to Learned ✓, as an uploaded file does.
There's one note per website, and saving replaces it — edit the text and save again to update. Keep it in one place and it stays easy to review. The counter under the field counts characters: at least 20 before it can be learned from, and up to 20,000 — several pages if you need them.
Good to know
- It behaves like an uploaded document. It counts against the same Knowledge Base storage as your uploads, shows the same statuses, and is deleted the same way — deleting it removes its text from the agent's knowledge immediately.
- Owner and partner only. In a website shared with you as an operator, the field isn't shown — only the owner, or a partner working on the site (see Team & Operators), writes it.
- If loading the saved note fails, editing locks and we tell you the note is still stored and in use. Reload the page; what you wrote is kept.
- If learning was interrupted, the note shows Error ✗ with a note to save it again — pressing Save and learn once more re-indexes the same text.
File upload
For knowledge that isn't on your public website — internal manuals, return policies, pricing sheets, product specs, FAQs, support transcripts, training material — upload it directly in Dashboard → Knowledge Base.
Uploading a document
- Go to Dashboard → Knowledge Base and pick the website in the workspace switcher at the top of the left-hand navigation.
- Drag a file into the drop-zone, or click browse to choose one.
- Accepted formats: PDF, DOCX, and TXT. Anything else is rejected with an inline message.
- Click Upload. Each file's status moves from Learning... to Learned ✓ once indexing finishes — usually under a minute. This updates live; you don't need to refresh.
If something goes wrong (an unreadable or empty document), the file shows an Error ✗ status with a short reason so you can fix and re-upload. In practice there are three causes:
- The file is corrupt — it didn't survive whatever produced or transferred it. Re-export it and try again.
- It's password-protected — we don't ask for the password, so the file stays unreadable to us. Save an unprotected copy and upload that.
- It's a scanned, image-only PDF with no text layer — a photographed or scanned document is a picture of words, not words. There's no OCR on uploads: re-export it as a text PDF, or convert it to DOCX.
Uploaded files are extracted, split into meaningful chunks, and indexed alongside your crawled pages.
Managing uploaded files
Uploaded documents are listed under Uploaded Documents with their name, size, and status. Delete a file with the trash icon — this removes both the file and its indexed content from the agent's knowledge (a confirmation step guards against accidents).
Read-only for operators. If you're working in a website that was shared with you as an operator (see Team & Operators), the Knowledge Base is view-only — the upload zone and delete buttons are hidden. Only the website owner, or a partner working on the site (see Team & Operators), manages knowledge files.
Storage and file size
A strip above the drop-zone shows your Used Storage (used / limit) and your Max File Size. Used Storage is measured against a single per-plan number (see the per-plan limits below): that figure is your total budget for the documents on that website combined, your typed note included. Max File Size is the largest single file you can upload — the smaller of that budget and 25 MB. Once your files add up to it, the next upload is rejected until you delete something.
One file larger than 25 MB is refused, whatever your plan's storage figure says. There's a separate ceiling on a single upload, and on a plan whose storage figure is above 25 MB it's the one you'll actually meet: a single 40 MB PDF is refused even when the storage budget has room for it. Split a very large document, or convert it to text — a text-only export of a big PDF is usually a small fraction of the original. This is a limit on one transfer, not on how much you can store in total; several files under 25 MB each can still fill a larger budget.
An upload bigger than your plan's storage figure, or one that would push your total over it, is turned down immediately without leaving your browser, and so is a file over the single-upload ceiling, such as a 40 MB PDF. When you're near the cap, either delete older documents or upgrade your plan for more room.
Your plan can also cap how many documents a website holds. Each uploaded file counts as one, and so does your typed note. One past the cap is refused and not stored; the notice gives the cap. Delete a document or move to a larger plan. If the notice says your plan includes no knowledge base files, deleting a document does not help, and only a plan that includes them lets you add one.
Note: file upload here is your knowledge base — the documents your agent learns from. It's different from Document Analysis, where visitors upload a file mid-chat (an invoice, a resume, a spreadsheet) for the agent to parse on the spot. That's a separate feature — see Document Analysis.
Plan limits at a glance
The three content limits below scale with your plan.
Automatic re-crawl frequency
The fastest re-crawl each plan permits:
| Plan | Starter | Pro | Business |
|---|---|---|---|
| Fastest automatic re-crawl | Weekly | Daily | Every 12 Hours |
These are ceilings rather than the schedule you get out of the box: a new site is set to daily and then clamped to the plan's ceiling, so on a plan whose fastest re-crawl is slower than daily the site runs at that ceiling, and otherwise it starts at daily. There's no self-service control for the cadence today — if your plan allows a faster schedule than daily and you want it, email support@novixagent.ai.
Crawl coverage (pages and depth)
How much of a large site each crawl pass covers:
| Limit | Starter | Pro | Business |
|---|---|---|---|
| Pages read per website | 500 | 2,500 | 10,000 |
| Crawl depth (links followed from the start page) | 2 | 3 | 4 |
Knowledge Base storage
Your total budget per website for what you add by hand — uploaded documents and your typed note together (the note is text, so its share is negligible). Remember the separate 25 MB ceiling on a single file, described above:
| Plan | Starter | Pro | Business |
|---|---|---|---|
| Knowledge base storage per website (MB) | 10 | 100 | 1,024 |
A single file also has to clear a fixed 25 MB ceiling, so where the storage figure above is under 25 MB the storage figure is the single-file limit, and otherwise the 25 MB ceiling is.
Need more room or a faster refresh? Upgrade your plan.
Turning knowledge gaps into content
The most valuable feedback loop lives in your analytics. Dashboard → Analytics → Messages (its plans are in Which tabs are on which plan) surfaces "Questions Without an Answer" — the individual questions your agent found no usable evidence for — alongside "Topics Without an Answer", the same gaps rolled up by theme. Treat an entry as a page to write or a document to upload.
Fix the gap once — add the answer to your site (picked up on the next crawl), upload a document, or simply type the missing line into "Anything we missed" for an answer that works within the minute — and the next visitor who asks the same thing can get a confident, cited answer. See Website Analytics for the full loop.
Have a tricky content edge case — a gated resource, a huge document set, an unusual site structure? Email support@novixagent.ai and we'll help you get it indexed.