Follow a company-policy question from the employee handbook to a checked answer. Each stage gives you a way to tell why a document assistant got something right or where it went wrong.

Example policy question

  1. Source. A fictional company keeps policies in a knowledge base. Its current Travel Policy §4 says, “For overnight work trips, employees may claim a standard rail fare with an itemized receipt.” Its Office Supplies Policy §2 also mentions receipts.
  2. Question. An employee asks, “Can I expense my train ticket for an overnight work trip, and what’s the deadline?”
  3. Retrieval. Keyword search can find “overnight” and “receipt.” Search using embeddings can bring “train ticket” close to “rail fare” in a shared representation. Encode the question and indexed passages into the same space, typically using the same embedding model and dimensions, so their scores are comparable. OpenAI’s embeddings guide shows how this works in semantic search. A vector database is one place to store and search those vectors. Hybrid search combines keyword and vector results. Here the right passage is Travel Policy §4. The office supplies passage shares a word but covers a different expense.
  4. Request context. The app supplies the question, the relevant passage and an instruction to cite the passage and flag any missing information. What the model can use for this answer is its context. This retrieval-and-answer pattern is retrieval-augmented generation (RAG). Keyword retrieval can also supply the passage, so embeddings and vector databases are optional.
  5. Example answer. “You can claim a standard rail fare for an overnight work trip if you have an itemized receipt (Travel Policy §4). This passage gives no claim deadline.” The answer is generated during inference, when the trained model runs on the supplied input.
  6. Check. Open §4 and confirm the fare type, overnight condition and receipt requirement. Look for a current expense filing policy before giving a deadline. If retrieval finds only the office supplies passage, report that the rail policy is missing from the retrieved material and ask where to find it.

The application controls who may access each document, which policy version is current, and when changes or removals reach the search index. Check those controls before supplying excerpts to the model. OpenAI’s retrieval guide describes search and filtering across indexed sources.

Match the change to the failure

  • Use prompting when the passage is available and the answer needs clearer instructions, such as requiring a section citation for each policy claim.
  • Improve retrieval when the app selects the wrong version, overlooks the policy or supplies an irrelevant passage.
  • Evaluate fine-tuning when a repeated task needs a behavior change that a good baseline still misses, such as routing reviewed cases into a stable set of categories. Additional training adapts learned parameters. Keep changing policy facts available through current sources. OpenAI’s model optimization guide describes evaluating a baseline and training changes.

Try the example with two passages you can inspect yourself. Check one question the passages answer and one detail they leave open. Judge passage selection and claim support separately, then use the prompting and fine-tuning comparison to choose what to change next.