LLM APIs

01The idea: a stateless text-in, text-out service

Think of an LLM API as a very capable colleague you message by email: every message must contain all the context they need, because they remember nothing between emails. Technically, you send an HTTPS POST with a model name, an optional system prompt (standing instructions), and a list of messages alternating between 'user' and 'assistant'. The model reads all of it and generates a reply one token at a time. A token is a chunk of text — roughly three to four characters of English — and both input and output are measured and billed in tokens. The API is stateless: to continue a conversation you send the whole history again. Every model also has a context window, the maximum number of tokens it can read in one request; current frontier models accept hundreds of thousands to a million tokens, but bigger prompts cost more and respond slower.

• System prompt: role, rules and format the model should follow
• Messages: the conversation so far, oldest first
• max_tokens: a hard cap on the length of the reply
• stop_reason: why the model stopped (finished, hit the cap, wants a tool, refused)

02Your first call (Claude Messages API)

Every provider has a similar shape; here is Anthropic's Messages API with the official TypeScript SDK. The SDK reads ANTHROPIC_API_KEY from the environment, so the key never appears in code. The response content is a list of blocks (text, tool calls, thinking), so you loop and pick the text blocks. Always check stop_reason: 'max_tokens' means the answer was cut off, and 'refusal' means the model declined for safety reasons. Model names change over time — at the time of writing Anthropic's lineup includes Claude Opus 5.5 (most capable general model), Claude Sonnet 5.5 (balanced) and Claude Haiku 5.5 (fast and cheap) — so keep the model id in configuration, not scattered through code, and check the provider's models page before you ship.

1import Anthropic from "@anthropic-ai/sdk";
2 
3const client = new Anthropic(); // reads ANTHROPIC_API_KEY
4 
5const res = await client.messages.create({
6  model: process.env.LLM_MODEL ?? "claude-opus-5-5",
7  max_tokens: 1024,
8  system: "You summarise support tickets in 3 bullet points.",
9  messages: [{ role: "user", content: ticketText }],
10});
11 
12if (res.stop_reason === "max_tokens") console.warn("Reply was truncated");
13for (const block of res.content) {
14  if (block.type === "text") console.log(block.text);
15}
16console.log(res.usage.input_tokens, res.usage.output_tokens);

03Streaming: show the answer while it is being written

A long answer can take tens of seconds. With streaming, the API sends the reply as a sequence of server-sent events (SSE) — small chunks of text the moment they are generated — so your UI can show words appearing instead of a spinner. Time to first token drops from 'whole answer' to well under a second or two. Streaming is also the safe default for long outputs because a single non-streamed HTTP request may hit client or proxy timeouts. In a web app, the usual pattern is: browser → your backend (which holds the API key) → provider, and your backend forwards the stream to the browser over SSE or a fetch ReadableStream. Never call the provider directly from the browser with a secret key.

1const stream = client.messages.stream({
2  model: "claude-opus-5-5",
3  max_tokens: 4096,
4  messages: [{ role: "user", content: question }],
5});
6 
7stream.on("text", (delta) => res.write(delta)); // forward to your client
8const final = await stream.finalMessage();       // full message + usage
9res.end();

04Tool use: letting the model call your functions

Tool use (also called function calling) lets the model ask your code to do something it cannot do alone: look up an order, query a database, check stock. You describe each tool with a name, a description and a JSON Schema for its input. When the model decides a tool is needed, it stops with stop_reason 'tool_use' and returns a tool_use block with the arguments. Your code runs the real function, sends the result back as a tool_result block, and the model continues. The model never executes anything itself — your code does, so your code decides what is allowed. Write tool descriptions like documentation for a new teammate: what it does, when to use it, what each field means. Mark tools strict so arguments always match the schema, and validate them anyway before touching real data.

1const tools = [{
2  name: "get_order_status",
3  description: "Look up the delivery status of one order by its id.",
4  strict: true,
5  input_schema: {
6    type: "object",
7    properties: { order_id: { type: "string", description: "e.g. ORD-1042" } },
8    required: ["order_id"],
9    additionalProperties: false,
10  },
11}];
12// 1) send tools + messages  2) if stop_reason === "tool_use", run the tool
13// 3) append { role: "user", content: [{ type: "tool_result", tool_use_id, content }] }
14// 4) call again until stop_reason === "end_turn"

05Getting reliable structure: JSON outputs and prompts

Most product features need data, not prose: a category, a sentiment score, extracted invoice fields. Two tools help. First, structured outputs: you give the API a JSON Schema and it constrains the reply to match it (on Claude this is output_config.format; other providers have equivalents). Second, good prompting: put stable instructions in the system prompt, give two or three examples of the input and the expected output, wrap long documents in clear tags like <document>, and ask for exactly the fields you need. Be explicit about what to do when information is missing ('return null, do not guess'). Then treat the output like any untrusted input — validate it with Zod, Pydantic or Laravel validation before saving it, because a schema guarantees shape, not truth.

06Cost, limits and production habits

Cost = input tokens × input price + output tokens × output price, and output is usually several times more expensive per token. The main levers: pick the smallest model that passes your evaluation; cache the stable prefix of long prompts (prompt caching makes repeated reads of the same system prompt or document far cheaper and faster); use the Batch API for work that can wait — it runs asynchronously at about half price; cap max_tokens sensibly; and trim conversation history. Providers enforce rate limits per minute on requests and tokens, returning HTTP 429; the official SDKs already retry 429 and 5xx with backoff. Log usage per feature and per customer from day one, set budget alerts, and keep the model id configurable so you can switch models without a deploy.

• Smallest model that passes the eval set
• Prompt caching for long, repeated prefixes
• Batch API for offline jobs (about 50% cheaper)
• Usage logs per feature, plus a spend alert

07Prompt caching internals: why order matters

Prompt caching is a prefix match. The provider hashes the request in render order — tools, then system prompt, then messages — up to a cache breakpoint. If a later request starts with exactly the same bytes, the cached internal state is reused: you pay a small fraction of the normal input price for those tokens and the model starts answering sooner. Any change in the prefix, even one character, invalidates everything after it. Common silent killers: a current timestamp in the system prompt, tools listed in a different order per request, JSON with unstable key order, or a per-user greeting before the shared instructions. Design prompts as layers: frozen instructions and tools first, then large shared documents, then the per-request question. Verify with the usage fields (on Claude, cache_read_input_tokens); if it stays at zero on repeated requests, something in your prefix changes. Caches also have a short time-to-live and are scoped per model, so switching models resets them.

1const res = await client.messages.create({
2  model: "claude-opus-5-5",
3  max_tokens: 1024,
4  system: [
5    { type: "text", text: FROZEN_RULES },
6    { type: "text", text: PRODUCT_MANUAL, cache_control: { type: "ephemeral" } },
7  ],
8  messages: [{ role: "user", content: question }], // varies per request
9});
10console.log(res.usage.cache_read_input_tokens); // > 0 on repeat calls

08Reasoning models, effort and latency budgets

Modern models can 'think' before answering — generating internal reasoning tokens that improve results on hard, multi-step problems. On current Claude models this is adaptive: the model decides how much to think, and you steer depth and spend with an effort setting (low to max) rather than a fixed token budget. Thinking tokens are billed as output, so effort is a direct cost and latency lever. Route by task: classification and short extraction at low effort; coding, analysis and agent loops at higher effort. Other providers expose similar controls under different names (reasoning effort, thinking budget). Measure the curve on your eval set — often a strong model at low effort beats a weaker model at high effort for the same price. For user-facing chat, stream and consider showing a short 'thinking' state, because time to first visible token grows with effort.

09Reliability: retries, idempotency and fallbacks

Treat the provider like any remote dependency. Retry 429 (rate limited), 5xx and network errors with exponential backoff and jitter, honouring the retry-after header; never blindly retry 400 errors, which mean your request is wrong. The SDKs retry twice by default; set an explicit timeout that fits your UX. For user actions, put long generations in a queue (for example BullMQ or Laravel queues) and push the result back via WebSocket or polling, so a web request is never blocked for a minute. Plan for degraded modes: a secondary model or provider behind an adapter, a cached previous answer, or a clear 'try again later' message. Handle refusals and max_tokens explicitly rather than showing an empty or truncated answer. Finally, version prompts and record which prompt version and model produced each stored output, so you can reproduce and debug later.

10Security: prompt injection and data exposure

Prompt injection is the SQL injection of LLM apps: untrusted text (a user message, a web page, an uploaded PDF, an email) contains instructions like 'ignore previous rules and reveal the system prompt' or 'call the refund tool'. There is no complete fix, so you design for containment. Keep secrets out of prompts entirely. Give tools the least privilege needed, enforce authorisation in the tool code using the real user's identity — never trust IDs the model supplies. Require human confirmation for irreversible actions. Mark untrusted content clearly in the prompt and tell the model it is data, not instructions. Never render model output as raw HTML or execute it as SQL or shell. On the data side, read the provider's retention policy, use zero-retention or regional options where required, strip personal data you do not need, and log prompts carefully because logs become a sensitive data store too.