• Module 3 · Teach Ace What You Know
  • 2 min read · about 20 min lab
  • Builds: Your knowledge vault in Ace

How to Test an AI Knowledge Base with "Only I Know This" Questions

Ask questions only your knowledge base can answer, and fix what it misses.

Key takeaway & what you'll build

Uploading a document doesn't guarantee Ace uses it, because the vault works by matching your message to passages. Test it with questions only your documents can answer. Fix misses by checking each document's status, splitting long files, using your customers' words, and deleting old facts.

You'll build: Your knowledge vault in Ace

Trust, but verify

Uploading a document doesn't guarantee Ace will use it when you need it. The vault works by matching: Ace compares your message to passages in your documents and brings in the closest ones. Test it the way you'd test anything important.

The "only I know this" test

Ask questions whose answers exist only in your documents:

  • "What's included in my 1:1 coaching package, and what does it cost?"
  • "What do I usually say when someone asks whether this works for beginners?"
  • "Summarize my point of view on posting frequency."

If Ace answers with your details, retrieval is working. If it gives a generic answer, one of these is usually why:

  • Generic answer — Likely cause: Document still processing, or failed · Fix: Check its status in the Knowledge panel
  • Right topic, wrong details — Likely cause: The key fact is buried in a long document · Fix: Split it into a short, focused document
  • Ace doesn't know a term — Likely cause: Your document uses different words than your question · Fix: Use the same words your customers use, in both
  • An old fact comes back — Likely cause: An outdated document is still in the vault · Fix: Delete it

If Ace warns that your knowledge base is unavailable, that's a temporary problem on our side, not your documents. Try again shortly.

Why this matters for voice

Your voice guide is a document too. A request like "write a post about pricing" will pull in your pricing material, but it may not pull in your voice guide, because that request isn't about voice. That's exactly why the must-follow parts of your voice live in your Brand Profile, which is always included, and not only in the vault.

Try it

Write two questions only your vault can answer, and what you expect Ace to say. Then ask them in Ask Ace. You'll submit the results in the lab.

Hands-on lab

Stock Your Vault

List at least three documents you added to Global Knowledge, with one line each on why it belongs there. Then submit two "only I know this" test questions, the answer you expected, what Ace actually answered, and what you changed (if anything) after seeing the results.

What the AI grader checks for

  • Relevant documents (40%): The documents are the learner's own, focused, and useful for their content (voice guide, offers, FAQs, best work), with a clear reason for each.
  • Real retrieval tests (40%): Both questions can only be answered from the learner's material, and expected versus actual answers are recorded honestly.
  • Reflection and fixes (20%): The learner identifies at least one gap or weakness and describes a concrete fix, or explains why no change was needed.

Ask the AI course tutor

The tutor knows this whole course. Pick a question to start a conversation about this lesson.