How to Build an AI Chatbot on Your Own Company Data

To build an AI chatbot on your own company data, you do not retrain a model. You pick a narrow scope, clean the documents that cover it, index them for retrieval, instruct the model to answer only from retrieved passages with citations, and test against a set of real questions before anyone uses it. The steps below are the order we follow as engineers, including the parts that are easy to skip and expensive to skip.

Key takeaways

• Scope first. One audience and one topic area beats "everything we know".
• Content quality decides answer quality. Budget real time for a content audit.
• Write the test questions before you build, so you know what "working" means.
• Decide what the bot does when it does not know: say so, and offer a way to reach a person.
• Launch to a small group, read the conversations, and fix the gaps they expose.

Where this guide fits: This guide covers a chatbot for customers trained on company data. For an assistant for your own staff read internal knowledge assistants.

Step 1: Define the job

A chatbot project fails most often at the scoping stage. Answer four questions in writing.

• Who asks? Customers on your website, or staff inside the company. This changes tone, risk and access rules.
• What do they ask? Pull the real questions from your support inbox, chat logs or the colleague everyone interrupts.
• What must it never do? Give legal or medical advice, quote prices that are not published, promise delivery dates.
• What happens at the edge? Where the conversation goes when the bot cannot help.

For example, a software company might scope its first bot to "answer setup and billing questions from published help articles, for logged-out visitors, and hand over to support for anything about a specific account".

Step 2: Audit and prepare the content

List every source that covers the scope and check each one.

1. Is it current? Archive superseded versions. Two refund policies in the index means two possible answers.
2. Is it the approved version? Drafts and personal notes should stay out.
3. Is the text extractable? Scanned PDFs need text recognition. Complex tables and diagrams may need rewriting as text.
4. Is anything missing? Compare the documents with the question list from step 1. Common questions with no written answer are a content gap, and no model can fill it honestly.
5. Who may see it? Mark each source public, internal, or restricted to a group.

This step is unglamorous and it is where most of the final quality comes from. Often the best outcome of the audit is a short list of documents someone needs to write.

Step 3: Write the test set

Before building, write down a set of questions, each with the correct answer and the document it comes from. Include easy questions, questions phrased the way customers really type, questions with no answer in your content, and questions the bot should decline. This set is your definition of done. It also stops the project from being judged by whichever question the most senior person happens to try.

Step 4: Build the retrieval pipeline

This is the technical core, usually called RAG. In short:

• Chunk each document into passages along its headings, keeping the title and section name with each chunk.
• Embed the chunks and store them in a vector index with metadata: source link, date, audience.
• Retrieve using meaning-based search, usually combined with keyword search for exact terms like product codes.
• Rerank the candidates and pass only the best few to the model.
• Filter by permission at retrieval time, so restricted passages are never fetched for the wrong user.

If these terms are new, what is RAG explains each one in plain language.

Step 5: Write the instructions and set the guardrails

The model needs clear, short instructions. The essentials:

• Answer only from the provided passages. If they do not contain the answer, say so.
• Cite the source for each answer, with a link.
• Stay within the defined topics. Politely decline the rest.
• Ask a clarifying question when the request is ambiguous, for example when two products share a name.
• Use the company's tone: length, formality, language.

Add checks in code as well, since instructions alone are not a control. Limit message length and rate. Treat text inside retrieved documents and user messages as content, never as commands, so that a line such as "ignore your instructions" has no special power. Keep secrets and internal system details out of the prompt entirely.

Step 6: Connect live data only where it is needed

Documents answer "how does it work". They do not answer "where is my order". If the scope includes account-specific questions, give the bot a small number of tools that look up live records through your systems' APIs, after the user is verified. Keep these read-only at first. A bot that can change a booking or issue a credit is an agent with real consequences, and it deserves its own design and approval steps.

Step 7: Evaluate, then launch small

Run the test set. For each miss, find the cause before changing anything.

Symptom	Likely cause	Fix
Right document exists, wrong passages retrieved	Chunking or search	Restructure chunks, add keyword search or reranking
Right passage retrieved, wrong answer	Instructions or model	Tighten the prompt, try a stronger model
Answers a question it should decline	No refusal path	Add a relevance threshold and an explicit "not found" response
Two different answers to the same question	Conflicting sources	Remove or reconcile the documents
No good answer possible	Content gap	Write the missing document

When the test set passes at a level you are comfortable with, release to a small group: internal staff, or a small share of website visitors. Read the conversations. Real users ask things you did not predict, and those questions go into the test set.

Step 8: Keep it alive

A knowledge bot is a product, not a one-off project. Someone needs to own it.

• Sync the index when documents change, and remove retired ones.
• Review unanswered questions regularly. They are a free list of what to document next.
• Collect feedback with a simple thumbs up or down and a way to report a wrong answer.
• Rerun the test set after any change to content structure, prompt or model.
• Watch usage and model costs so there are no surprises.

Mistakes we see most often

• Indexing the whole shared drive. Old drafts and duplicates poison the answers.
• No test set. Quality becomes a matter of opinion, and every change is a gamble.
• Forcing an answer every time. A bot that never says "I don't know" will invent.
• Hiding the sources. Without links, users cannot check, and one wrong answer ends their trust.
• No handoff. A customer stuck in a loop with a bot is worse off than with no bot.

Related guides

• Internal Knowledge Assistants: Give Your Team Instant Answers
• What Is RAG? Retrieval-Augmented Generation Explained
• RAG vs Fine-Tuning: Which Should You Choose?
• Build vs Buy: Custom AI Agents or Off-the-Shelf Tools?
• Service: AI Chatbots and Knowledge Assistants

Frequently asked questions

Do we need to train or fine-tune a model on our data?

Usually not. Retrieval gives the model your content at question time, which is easier to update and lets it cite sources. Fine-tuning is occasionally added later to shape style or output format.

What formats can the chatbot use?

Anything that can be turned into clean text: web pages, PDFs, office documents, help centre articles, wiki pages. Scans, images and complex tables need extra processing and should be tested carefully.

Is our data safe?

That depends on how it is built. Check where the index is hosted, what the model provider's terms say about using your data, who can query which documents, and what is logged. These are design decisions, not afterthoughts.

Can the same chatbot serve customers and staff?

It can share the same pipeline, but the two audiences should have separate permissions and often separate instructions. Internal documents must be filtered out for public users at retrieval time.

Sources and further reading

Primary documentation and research behind this guide.

• Lewis et al.: Retrieval-Augmented Generation for knowledge-intensive NLP tasks
• pgvector: vector search for PostgreSQL
• OWASP: Top 10 for large language model applications

Building a chatbot on company data is mostly disciplined preparation: scope, content, tests, then code. Growmerz builds these under our AI chatbots and knowledge assistants service. To work out the right scope for a first version, book a free AI audit.