Practice: How LLMs work inside applications: tokens, context windows and model APIs
Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.
Main tool: OpenAI Responses API · The Responses API returns token usage, cached token counts and streaming events on every call, so each idea in this lesson can be measured from a laptop rather than taken on trust. · Alternatives: Anthropic Messages API (a similar stateless request shape with explicit cache breakpoints and a required output token cap), Amazon Bedrock, Google Vertex AI and Azure OpenAI (the same model families behind cloud account authentication, quotas and regional endpoints), LiteLLM (an open source library and proxy that puts one request format in front of many providers), OpenRouter (a hosted gateway that routes one key to many models), and tiktoken (a local tokenizer for counting tokens offline, not a model API).
Every step links into the tool's own documentation; links were checked on 2026-10-11 · version 1
Setup (one time)
1. Install the OpenAI CLI on your laptop with the package installer the section names.
The CLI lets you send Responses requests and save their output without writing an application first, so the early steps stay about the model and not about client code.
Explained in: OpenAI CLI (developers.openai.com), checked 2026-10-11
Done when: Asking the CLI for its version prints a version number in your terminal.
2. Create an API key and authenticate the CLI with it through an environment variable; this API is metered, so it needs a small prepaid credit and has no free local substitute.
Every request is authenticated and billed per token, which is the cost model this lesson teaches. Using the smallest model keeps the whole path to a few cents.
Explained in: OpenAI CLI (developers.openai.com), checked 2026-10-11
Done when: The CLI accepts an authenticated call without an authentication error and the key is not written into any file you would commit.
3. Choose the smallest general model in the lineup as your default for this path and write down its exact model id.
Model choice sets the price per token, the speed and the context window size, so you need one fixed baseline before you compare anything.
Explained in: Model selection | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: You have one model id noted, with a one-line reason for choosing it over the larger models.
Steps
1. Run a first Responses API request with your chosen model id and a one-sentence prompt, then read the whole response object and not just the text.
The first run shows that a call is one stateless request in and one response object out, with the generated text and a usage block side by side.
Explained in: Using GPT-6 | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: You can point to the output text, the model id and the input and output token counts in the returned response.
2. Configure the request parameters the section lists, such as reasoning effort, text verbosity and the output token cap, changing one at a time on the same prompt.
Parameters are the only controls an application has over a model call. Changing them one at a time shows which ones alter length, cost and behavior.
Explained in: Using GPT-6 | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: You have a small table with one row per parameter change showing the output length and token usage it produced.
3. Compare the input token count of a verbose prompt against a trimmed prompt that asks for the same thing.
Models read tokens, not characters or words, and you pay and wait per input token. Seeing the count move as you edit text makes tokenization concrete.
Explained in: Latency optimization | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: The usage block reports fewer input tokens for the trimmed prompt and you can state the rough ratio of tokens to words for your text.
4. Run the same prompt once with a tight output cap and once asking for a long answer, timing each call.
A model produces its answer one predicted token after another, so generation time grows with the number of output tokens. This is the practical consequence of next-token prediction.
Explained in: Latency optimization | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: Your timings show the long answer taking clearly longer, roughly in proportion to its output token count.
5. Run a five-turn conversation twice, once resending the full history as input each turn and once chaining turns with the previous response id, recording input tokens per turn.
The API keeps no memory between calls unless you ask it to, so every turn re-sends or re-loads the earlier turns into the context window. Watching input tokens climb shows the window is a budget that fills up.
Explained in: Migrate to the Responses API | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: You have per-turn input token counts that rise every turn in both variants and an estimate of how many turns fit before the window is full.
6. Enable streaming on a long-answer request and record the time to the first text event and the time to the final event.
Streaming delivers tokens over server-sent events as they are generated, which changes how fast the answer feels without changing total generation time.
Explained in: Streaming API responses | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: Text appears incrementally in your terminal and your notes show the time to first token is much shorter than the time to completion.
7. Write your instructions and prompts as versioned files in a local git repository and compare the outputs and usage of two prompt versions on the same input.
A prompt is part of the application and changes its cost and behavior, so it needs the same history and comparison as code.
Explained in: Text generation | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: Your repository holds two committed prompt versions and a saved comparison of their outputs and token counts.
8. Package each experiment as a request body file and run it through the CLI instead of typing the parameters each time.
A request body file is the complete, portable definition of a model call: model, input, parameters. It is the artifact you hand to a teammate or promote from an experiment to a service.
Explained in: OpenAI CLI (developers.openai.com), checked 2026-10-11
Done when: Each earlier experiment can be rerun from a committed request body file with one CLI call and gives the same kind of result.
9. Run two requests that share a long identical instruction prefix and differ only at the end, then restructure a prompt so its stable content comes first.
Prompt caching matches on the prefix of the request, so the order of content decides whether repeated context is billed and processed at the cached rate.
Explained in: Prompt caching | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: The usage block of the second request reports a non-zero cached token count and the second call returns faster than the first.
10. Run the prompt cache diagnostics on a request that misses the cache, fix the cause it reports and rerun to confirm the hit.
Cache misses are a common hidden cost in the field, usually caused by a changing value near the start of the prompt. The diagnostics tell you where the prefix diverged instead of leaving you to guess.
Explained in: Prompt cache diagnostics | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: The diagnostics show a miss with its cause before your change and a hit with cached tokens after it.
11. Create a script that reruns all your saved request body files against an explicitly pinned model id and fails when a response is missing or its token usage leaves the range you recorded.
Pinning the model and replaying fixed requests makes cost and behavior reproducible, and the same script can run in CI before any prompt or model change ships.
Explained in: Models and providers | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: The script passes twice in a row on a clean checkout and fails when you deliberately change the model id or inflate a prompt.
12. Write a cost estimate for a production conversation using your measured input, cached and output token counts per turn, then compare the estimate with the usage you actually logged.
A production setup needs a per-conversation cost budget before launch and a check against real usage after it. The section frames the arithmetic for realtime sessions, and you apply the same per-token accounting to your logged Responses usage.
Explained in: Cost optimization | OpenAI API (developers.openai.com), checked 2026-10-11
Done when: You have a one-page estimate of cost per conversation and per thousand conversations, with the gap between estimate and logged usage explained.
Back to How LLMs work inside applications: tokens, context windows and model APIs