Suppose a team wants an AI assistant to turn a long project review into five risks with section references. Seven terms help you work out what fits, what can get cut off, how usage is billed and where the wait happens.
Follow one request from the document to the risk list
- The API is how the team’s app sends the review and instructions and receives the result. Its request options determine which model and output controls the app can use.
- The model processes the request. Choose one that can handle the input size and check how well it finds risks and cites sections on a short reference case.
- A token is a unit the model processes. The review, the brief and the response contribute to usage. Token count can differ from word count and says little about the quality of the risk list.
- The context window is the model’s capacity for a request. The review has to fit alongside the instructions, any included history and room for generation under the model’s accounting rules.
- A max tokens setting caps generated output under the API’s rules. A request can fit inside the window and still run out of output budget before the fifth risk.
- Inference is the model processing the supplied review and generating a response. It uses a trained model with the input available for that request.
- Latency measures delay. The team may first see text, then receive the whole risk list, then wait for the app to validate the references.
Hypothetical token budget
Assume a 20,000-token context window whose accounting includes input and generated tokens. The review uses 14,000 tokens, the brief and included history use 2,000, and message framing uses another 500. That’s 16,500 input tokens. Set a generated-token cap of 2,000, giving a budget of 18,500 tokens and 1,500 tokens of room inside the window.
If the service counts 700 reasoning tokens against that 2,000-token cap, up to 1,300 tokens remain for visible output. Check your API’s usage and completion fields. Models and endpoints can account for reasoning, message framing, tool schemas and generated output differently. OpenAI’s reasoning guide and Responses API reference explain their accounting and output cap.
Billing depends on the service’s rates and reported usage. Input and output can have separate rates, and eligible cached input may have a separate rate. A cached prefix can still occupy part of the request budget. OpenAI’s prompt caching guide describes one provider’s cached-input category.
Hypothetical timeline
Start the clock when the app sends the request. The first visible token arrives at 2 seconds, the full streamed response arrives at 8 seconds and the app finishes validating references at 10 seconds. The team sees progress after 2 seconds and can use the checked list after 10. OpenAI’s latency guide describes how streaming affects what people see while output is still arriving.
Choose a fix for the limit you hit
If the input is oversized, remove irrelevant history or work section by section. Keep section identifiers and combine the risks in a final pass. If output is truncated, check the completion status, ask for a shorter format, split the deliverable, or raise the cap if there is room. Before sharing the result, open each referenced section and confirm the passage supports the named risk.