Ask Your Company Documents Anything: What a Knowledge Assistant Is and How to Check Its Answers

A company knowledge assistant is supposed to stop made-up answers by answering only from your own documents. A peer-reviewed 2025 study tested three commercial legal research tools built that way, which their makers had marketed as avoiding or eliminating hallucinations, and found that they produced false information between 17% and 33% of the time.¹ That was less often than a general-purpose chatbot, and still far from zero. The study covered legal research queries, so read it as a finding about one field. It is also the reason the test at the end of this article exists.

A company knowledge assistant answers only from your own documents, cites the page it used, and says so when your documents do not cover the question. This article explains what that means in practice, what an assistant like this needs from you, and a four-part test you can run before your team relies on one. If you would rather talk it through, a free 30-minute Strategy Call with an AI adoption specialist ends with the first two or three workflows worth automating.

01. When the answer lives in one person's head

The scene you may already know. A COO asks the controller, for the third time this month, what the approval limit is for a certain kind of purchase. An estimator messages the senior estimator to ask which spec section governed the last job like this one.

The answer exists in a document. Nobody can find it faster than asking the person who remembers it. If you are that person, you already know how this feels.

The cost of looking. Gartner surveyed 4,861 full-time employees who use digital technology for work, at organizations with 100 or more employees in the U.S., U.K., India and China, between September and November 2022. It found that 47% struggle to find the information or data they need to do their jobs effectively, in a workplace where the average desk worker used 11 applications, up from six in 2019.² That is a survey of digital workers in four countries, not of any one kind of company, and Gartner sells research and advisory services. The signal is still clear: finding the right information is a daily cost, not an edge case.

The contractor version. In 2023, Deloitte Access Economics surveyed about 1,275 people in 12 countries for a data report commissioned by Autodesk, which sells construction software. It found that construction managers and executives spend an average of 11.5 hours a week researching and analyzing data, and that people in BIM/VDC and preconstruction roles spend more, at 12.3 and 13.4 hours.³ That is time spent researching and analyzing data, which is wider than looking for a document, so read it as a sign of the load, not a count of searches.

The continuity risk. The person everyone asks is also, one day, the person who leaves. BuildForce Canada's July 2026 outlook projects that 135,000 residential construction workers will retire by 2035, which the Canadian Home Builders' Association says is more than a fifth of that sector's current labour force. It also projects that as many as 92,000 workers across Ontario's construction industry will leave through retirement over the same period.⁴ Those are labour-force projections for one industry, not figures about your firm. Where knowledge sits only in people's heads, it leaves when they do.

What follows. A company knowledge assistant is one way to take the pressure off that person, if it is built and tested well. The rest of this article covers what it is, why a citation helps and does not prove anything, what to put in first, who may see what, and how to test one.

02. What a company knowledge assistant is, and what it is not

The definition, plainly. A company knowledge assistant is AI that answers only from your company's own documents, such as policies, procedures, contracts, past proposals and specs. It cites the page it used, and it says so when the documents do not cover the question.

In practice, the system first finds the relevant passages in your documents and then writes the answer from those passages only. The technique is called retrieval-augmented generation, or RAG. Lewis and colleagues described it at NeurIPS in 2020. The paper's abstract notes that large language models are limited in how precisely they can access and manipulate knowledge, and that providing provenance for their decisions was an open research problem. The abstract also notes that pairing a model with an explicit index of documents was a way to address it.⁵

That paper is six years old, so it is cited only to name and define the technique. The short version is one paragraph in the introduction to AI adoption. This article is the longer version.

Versus a general chatbot. A general-purpose model answers from what it learned in training. The U.S. National Institute of Standards and Technology (NIST) calls the confident presentation of false content "confabulation" and says it is a natural result of how generative models are designed, because they generate outputs that approximate the statistical distribution of their training data.⁶ A general chatbot has no reason to know your approval limits, your standard terms or last year's proposal. It will still answer.

Versus search, and versus "chat with a PDF." Search returns files, not an answer, and it does not tell you when nothing matches. "Chat with a PDF" works on a document you upload, with no owner, no permissions and no record of what was loaded. A company knowledge assistant is the governed version: a chosen set of documents, with owners, permissions and a record of what it used. That is a working distinction, not a standard.

Three behaviours to look for. These three come from the definition above, not from a standard.

1. It answers only from the documents you chose.
2. It shows the document and the page.
3. It says so when your documents do not cover the question.

If any of the three is missing, you have a different tool, and the rest of this article does not apply to it.

Where we use one. We use one daily at Houseblend, on our own documentation. Our company knowledge assistant service is the Build-stage version of the same idea for your documents.

03. Why "cites the page" matters, and why it is not proof

The citation makes checking possible. It is the feature that lets a person verify an answer in seconds. The research on citations explains why it is not the same as being right.

The legal study in detail. Magesh and colleagues, in the Journal of Empirical Legal Studies in 2025, ran what they describe as the first preregistered evaluation of AI-driven legal research tools. They found the makers' claims about hallucination-free results overstated, hallucinations reduced compared with a general-purpose chatbot, and substantial differences between tools in responsiveness and accuracy. They also proposed a typology for separating hallucinations from accurate answers.¹

The range quoted here, 17% to 33%, is the one in the journal's abstract. Stanford Law's own page on the same study says "more than 17%," so the journal wording is the safer one.

The citation studies. Liu, Zhang and Liang audited four public generative search engines that answer with in-line citations and published their results in 2023. On average, 51.5% of generated sentences were fully supported by citations, and 74.5% of citations supported the sentence they were attached to.⁷

In 2025 the Tow Center for Digital Journalism at Columbia tested eight generative search tools in February. It ran 1,600 queries in which each tool was given an excerpt from a news article and asked for its headline, publisher, date and URL. Collectively, the tools gave incorrect answers to more than 60% of queries. The Tow Center also found that the tools were generally bad at declining to answer when they could not answer accurately, offering incorrect or speculative answers instead. More than half of the responses from two of the eight cited fabricated or broken URLs.⁸

Both studies tested public web tools on tasks that are not your documents, and the Tow Center says its findings should not be extrapolated to all models. The use here is narrow: a citation next to an answer does not show the page says what the answer says, and a tool that never says "I don't know" is a tool that guesses.

A wrong answer can arrive with a citation attached.

NIST's Generative AI Profile warns that outputs may include confabulated logic or citations that purport to justify the answer, which may further mislead people into inappropriately trusting the output.⁶ The Open Worldwide Application Security Project (OWASP) lists overreliance, meaning users placing excessive trust in model output without verifying it, as a factor that makes misinformation worse. It names retrieval-augmented generation as one mitigation, and cross-checking with human oversight as another.⁹ Both point to the same conclusion: the citation is where checking starts.

Who answers for it. In Moffatt v. Air Canada, decided by the Civil Resolution Tribunal of British Columbia in February 2024, the airline argued it was not responsible for what its website chatbot told a customer. The tribunal called that "a remarkable submission" and held the company responsible for all the information on its website.¹⁰ That was a customer-facing chatbot on a public website, not an internal assistant, so only one point carries over: the company answers for what its tool says. Canada's privacy regulators put it for organizations using generative AI this way: accountability for decisions rests with the organization, not with an automated system.¹¹

Why the third behaviour matters most. An assistant that says "your documents do not cover that" is doing its job. One that fills the gap is doing the thing this article exists to catch. Which answers still need a named person to check them is a separate question, covered by the sort in which tasks AI drafts, which it runs, and which always need a person.

04. What to put in first, and who owns each document

Start with the questions, not the folders.

The documents an assistant draws on are the ones listed in the definition: policies, procedures, contracts, past proposals and specs. Which of them to load first depends on the questions people already ask the senior person. Begin with the documents that answer those. That is a method, not a standard, and no source below endorses it.

A starter list by role.

• Owner or CEO: the delegation-of-authority policy and standard terms.
• COO or head of operations: standard operating procedures, onboarding material and subcontractor or vendor terms.
• CFO or controller: the accounting policy manual, approval limits, the month-end checklist and contract templates.

By sector:

• Professional services: past proposals and engagement letters are often where the most repeated questions live.
• Construction and specialty contracting: specs, scope sheets and subcontractor agreements do the same job.
• Finance teams: policy manuals and close checklists do.

See AI for professional services, AI for contractors and AI for finance teams.

One current version, one owner. OWASP's guidance on vector and embedding weaknesses names knowledge conflicts that arise when data from multiple sources contradicts itself. It tells builders to accept data only from trusted and verified sources and to regularly audit the integrity of the knowledge base.¹² In plain terms: if two versions of a policy sit in the library, the assistant can cite either one.

Name one person who owns each set of documents. That is a rule of thumb, not a standard. ISO 30401 is the international standard for knowledge management systems, and ISO describes its requirements as applicable to any organization regardless of type or size.¹³ The standard text is paywalled, so only that description is cited. Checking that documents are current and owned is the same work as the "as of" and owner checks in the data readiness checks.

What not to load yet. Documents that hold confidential client information, personal information, or anything you could not show a new hire. CPA Ontario's guidance says confidential, restricted or personally identifiable information goes into AI tools only where the environment is verified as secure and compliant, under its Rule 208.¹⁴ Canada's privacy regulators say organizations using generative AI must comply with applicable privacy law and should limit the use of personal information to what is needed.¹¹ Those are professional and regulatory statements, not legal advice.

On NetSuite? If your documentation lives in NetSuite, Houseblend's NetSuite practice is separate, and its agentic delivery page is the place to start.

05. Who may see what, and how it stays current

Ask the question before anyone loads a document.

Does the assistant only draw on documents the person asking is allowed to open? That is a question to ask any vendor or partner, including us, not a claim we make for every tool. OWASP's 2025 entry lists unauthorized access and data leakage as a risk where access controls are inadequate or misaligned, and cross-context leaks where different groups share one store. Its mitigations start with fine-grained access controls and permission-aware stores, and include detailed logs of what was retrieved.¹²

Four questions to ask. These are working questions, not a source's.

1. Which documents will it see?
2. Who is allowed to see each one today?
3. Will it show a person only what that person could already open?
4. Can you show me which pages it used for a given answer?

Picture the owner's board notes and the HR policy library in the same store, and a question from the wrong person. That is the case permissions exist for.

Prompts and answers are data too. The Canadian Centre for Cyber Security advises avoiding personal information or sensitive corporate data in queries or prompts.¹⁵ Canada's privacy regulators add that where personal information must go into a prompt, it should be done only where authorised, and that retention schedules should cover prompts and outputs.¹¹

Keeping it current. A policy changes on Monday and the assistant still cites last quarter's. OWASP names the underlying problem: a model can fail to supersede old knowledge it learned in training with new data it retrieves.¹²

The practical rules are working rules, not a standard.

• The owner updates the document first.
• The old version is retired, not left beside the new one.
• Any change triggers a re-test, described in the next section.

The Government of Canada's guide on generative AI describes itself as a living document updated regularly,¹⁶ and the same idea applies to your library. For the dating habit that makes this easy, see the "as of" check in the data readiness checks. Who decides what an assistant may see is also a policy question, and AI policy and governance is the service that covers it.

06. How to test one before you rely on it

The frame. The legal study built a preregistered set of questions and a typology for telling hallucinations from accurate answers.¹ The idea scales down to an office. Collect real questions, record what the assistant said, and give each answer a verdict. This is a working method, not a standard.

Test 1: ask the questions people already ask. Collect 20 or so questions that people really put to the senior person in the last month, not questions the builder thinks are good. Twenty is a suggested starting size, not a benchmark.

Write down the right answer and the page it comes from before the assistant is asked. Then mark each response right, wrong or incomplete. This is the "judge first, then look" habit from the AI policy article, applied here.

Test 2: open every cited page. For each answer, open the page it cites and read the passage. Liu and colleagues found that only 74.5% of citations supported the sentence they were attached to.⁷ Mark every answer where the page does not say what the answer says. A citation that exists but does not support the answer is a failure, not a pass.

Test 3: the "not covered" probe. Ask five questions that you know your documents do not answer. The right response is a plain statement that the documents do not cover it. A confident answer is a failure.

The Tow Center found that the tools it tested were generally bad at declining when they could not answer accurately,⁸ and NIST describes confabulation as the confident presentation of false content.⁶ Ask any vendor or builder to show you this test.

Test 4: change something and ask again. Update one policy, retire the old version, and ask the same question. The answer should change and cite the new page. If it still cites the old one, the library or the pipeline is not current, and you are back at section 05.

Who does what, and what counts as ready.

• The owner or CEO decides what an acceptable "not covered" rate looks like for each set of documents.
• The COO owns the operational questions and who may use it.
• The CFO or controller owns the finance, contract and approval-limit questions.

Which answers still need a named person to sign off is the sort in the AI policy article. The Government of Canada's guide says to always review content generated by AI, even if the system seems to be reliable.¹⁶ That guide is written for federal institutions, so treat it as a worked model.

Write the bar down before the tests, not after. For example:

• No wrong answers on the approval-limit and contract questions.
• Every "not covered" probe declined.
• Every cited page opened.

Where the bar is missed, fix the library or the setup and run the tests again.

Where this fits in the series. This is the fourth article in our AI adoption series. The first, our introduction to AI adoption, explains Plan, Build and Run. The second, is your data ready for AI, gives five checks on the numbers behind an AI answer. The third, which tasks AI drafts, which it runs, and which always need a person, sets the rules for who signs off.

Where to read next. How we run weekly client status reports with AI agents walks through one Friday run, from hours and tickets to an email draft that a person reviews and sends.

This content may be AI-generated and may contain inaccuracies. It is not professional, legal, or financial advice. Verify independently before relying on it.