Use a combination of: automated metrics (BLEU/ROUGE for summarisation, pass@k for code), LLM-as-judge (a second model scores outputs against a rubric), human preference rating, and task-specific golden-set tests. Track all three over time — no single metric is sufficient. Log all production inputs/outputs for offline analysis.
Implement exponential backoff with jitter for retries, use a queue to smooth traffic spikes, cache identical requests where appropriate, and consider multiple provider fallbacks (e.g. primary: Claude, fallback: OpenAI). Monitor API latency and error rates with alerting.
Profile token usage per request, cache embeddings and repeated completions, use smaller/cheaper models for simpler subtasks, apply prompt compression techniques, set hard max_tokens limits, and implement usage quotas per user/team. A cost dashboard with per-feature breakdowns is essential.
Treat prompts as code: store them in version control, use a prompt management tool (LangSmith, PromptLayer, Langfuse) to track versions and run A/B tests, add tests that run on every prompt change, and document the intent and expected behaviour of each prompt.
You need: LLM-specific tracing (spans per model call with token counts, latency, model version) via Langfuse, LangSmith, or Arize Phoenix; standard infra monitoring (CPU/GPU, memory, API error rates); cost dashboards; and a way to log and search production prompts and completions for debugging. OpenTelemetry is increasingly standard for the trace layer.
Use the model provider’s streaming API (Server-Sent Events). On the backend, open an SSE or WebSocket connection and forward tokens as they arrive. On the frontend, consume the stream and append tokens to the UI progressively. This dramatically reduces perceived latency — users see the first word in ~500ms instead of waiting 10 seconds.
