Video & Podcast

Your fine‑tuned model is ready. Evals look good. Now you need to actually deploy it.

This is the part most guides skip: how do you take a checkpoint and turn it into a service that real users can hit? What infrastructure decisions matter? What boundaries do you set? What happens when the GPU runs out of memory on New Year’s Eve?

Think of it like opening a restaurant. Your recipes work great in the test kitchen. But opening night means: multiple orders arriving simultaneously, ingredients running low, equipment breaking, customers ordering things you never tested together, and kitchen staff who’ve never worked a real dinner rush.

The recipes don’t change. The system around them determines whether they survive service.

This post is about setting up the system that will run your model in production. It’s the steps between “model is done” and “users are hitting an endpoint that won’t fall over.”

Why Fine‑Tuned Models Are Different

If you’ve only built apps using hosted APIs (like Gemini, OpenAI, Anthropic, etc.), deploying your own fine‑tuned model feels deceptively familiar until it isn’t. The code to call the model looks identical. You send a prompt, you get a response. But the responsibility has shifted entirely.

Someone Else’s Kitchen vs. Your Kitchen

Using a managed API is like ordering takeout. You place an order, and food arrives. You don’t care if the oven broke or if the chef is overwhelmed; that’s the provider’s problem. They handle things like:

  • Batching: Grouping multiple orders to maximize efficiency.
  • Memory Safety: Preventing crashes when too many orders arrive at once.
  • Cold Starts: Keeping the kitchen warm so the first order is fast.

When you deploy a fine‑tuned model yourself, you’re now running the whole restaurant (system).

Fine‑Tunes Are More Specialized

You’ve narrowed the menu on purpose. That improves quality for your target dishes, but it also makes the model more brittle:

  • Outsized Effects: Small changes in prompt formatting (”preparation”) can cause massive quality drops.
  • Distribution Shift: Ingredients that don’t match your training data (unexpected inputs) cause bigger problems than they would for a generalist model.
  • Edge Cases: Orders you didn’t practice come out worse.

This is why versioning, boundaries, and observability matter more once you fine‑tune. You optimized for specific dishes served a specific way; production introduces a chaotic reality that training never simulated.

The Production Mindset Shift

Now that you own the infrastructure, your definition of success changes. Training optimizes weights. Production optimizes behavior under constraints.

In your test kitchen, you made perfect dishes one at a time. But in production, you aren’t cooking one dish at a time. You are dealing with:

  • Latency budgets - Customers expect food in minutes, not whenever the kitchen gets to it
  • Memory limits - The stove only has so many burners; try to cook everything at once and nothing works
  • Concurrent orders - Multiple tables ordering simultaneously
  • Unexpected requests - Someone orders off-menu or has an allergy you didn’t prep for
  • Cost constraints - Every minute of cook time and every wasted ingredient costs money

A model that’s impressive in evals can become unusable in production for the same reasons a perfect test dish can fail during a dinner rush. The recipe didn’t change. The conditions did. The question isn’t just “does it work?” It’s “does it work when 50 people show up at once?”

Right‑Sizing Your Deployment

Not every restaurant needs the same kitchen infrastructure.

A food truck with a simple menu and predictable hours can run lean. But if you’re serving 200 people a night, taking reservations from strangers, changing the menu weekly, and planning to expand? You need to set up systems that support your work. Ticket tracking, inventory management, kitchen protocols, backup plans for when equipment fails.

When Simple Is Enough

If your model is internal‑only, low‑traffic, non‑critical, and easy to roll back manually, a lightweight setup works: manual scripts, stdout logging, and ad-hoc rollbacks. Think: Controlled test dinner with friends.

When You Need More

The bar moves when external users depend on outputs, costs start to matter (margins are tight), the menu (prompts/RAG) changes weekly, or kitchen staff rotate (multiple people touching system). At that point, you’re no longer just serving a dish, you’re running a kitchen that has to work consistently whether you’re there or not.

What it Costs

With managed APIs, you pay per token, like ordering à la carte. If you send zero requests, you pay zero dollars. With your own GPU, you pay per hour whether you use it or not like paying rent on the kitchen. An A100 can cost roughly $1–$5+/hour (depending on provider and commitment) whether you serve 10,000 requests or sit idle.

This shift catches teams off guard:

  • High, steady traffic: You saturate the GPU. Cost-per-token drops well below API rates. The economics work.
  • Low or sporadic traffic: The GPU sits idle most of the time. You might pay $3/hour for a system that serves 10 requests. That’s $0.30 per request which is far more than any API would charge.

If your traffic is unpredictable, consider serverless GPU options (RunPod, Modal, Replicate) where you pay for active seconds rather than 24/7 rental. You’ll pay more per-request than a fully utilized self-hosted setup, but you won’t burn money on an empty kitchen.

Making This Concrete: A vLLM Deployment

Let’s walk through a simple, single‑node vLLM setup. Not because everyone should deploy this way, but because simple architectures fail in understandable ways. Complex ones just fail mysteriously. Below are eight deployment essentials. You won’t tackle them sequentially because some happen in parallel but you need all of them.

Assumptions

  • Single GPU: (A100 / H100 / similar) — We are skipping distributed inference for now to minimize complexity.
  • Containerized: You are running this in Docker or a similar isolated environment, not raw on a dev machine.
  • Compatible Model: Your fine-tune is based on an architecture vLLM supports (Llama, Mistral, Qwen, etc.).
  • Ready to Ship: You are past the “just testing” phase.

Why vLLM?

We use vLLM here to illustrate these principles because it is the current standard for open-weight serving, but the principles apply whether you use TGI, TensorRT-LLM, or llama.cpp. vLLM hits a sweet spot:

  • Fast Time‑to‑First‑Token (TTFT): Minimizes the “thinking” pause between user pressing enter and the first word appears.
  • Efficient Batching: Uses PagedAttention to manage memory like an operating system, fitting more requests onto the same GPU.
  • Predictable Memory: Prevents the dreaded OOM (Out of Memory) crashes by reserving space upfront.
  • Flexible: Works as a standalone server or as the engine inside heavy-duty infra like NVIDIA Triton or Ray Serve.

It’s not the only option, but it’s the best default.

The following are core steps that you want to apply when deploying your model inference server. These are not all the things you do when launching but they are some of the most crucial points. Now let’s walk through the eight practices that turn this server into a production system:

1: Stack Determinism (Pin Your Stack)

“Nothing changed” is only true if your stack cannot drift.

Pinning means locking to exact versions that are not “latest” or “>=4.0” but specific version numbers. If you run pip install vllm three weeks from now and it grabs version 0.5.0, your model might behave differently. It’s not because the weights changed, it’s because the inference engine did.

Create a requirements.txt that locks in these (and apply your own numbers for the xxxs):

vllm==X.X.X
torch==X.X.X
transformers==X.X.X

Also pin at the infrastructure level by using a container (Docker) or VM image that specifies CUDA version, Driver version and Base image.

2: Behavioral Consistency (Lock Generation)

Exploration belongs in evals, not live traffic.

Just as you pin your software libraries, you must pin your model’s behavior. Letting clients experiment with temperature=2.0 or max_tokens=4096 is how behavior drifts to be completely different without anyone noticing.

python
# In your config or wrapper layer
DEFAULT_GENERATION_CONFIG = {
    "temperature": 0.7,      # Controls randomness
    "top_p": 0.9,            # Nucleus sampling threshold
    "max_tokens": 512,       # Maximum response length
    "stop": ["</s>", "\n\n"], # When to stop generating
}

These are common parameters, but your model might need different values or additional settings. The key is: whatever worked in testing becomes your locked default. Also do not let every client choose their own parameters in production. Override only when there’s a strong, explicit reason.

3: Artifact Separation (Immutable Files)

You need to define a file structure that separates the things that change often (prompts) from the things that change rarely (weights). In a restaurant, you wouldn’t store your recipes in the same filing cabinet as tonight’s orders

The golden rule: Model weights, prompt templates, and inference config should be versioned independently.

  • Weights are heavy (GBs) and change monthly.
  • Prompts are light (KBs) and might change daily or much longer.
  • Configs define how the engine runs.

If you lump these together, you can’t fix a typo in a prompt without reloading a 50GB model.

The Production Directory Structure
Organize your artifacts like this:

/app/
  ├── models/my-finetuned-v3/    # IMMUTABLE & LOCAL (Weights)
  │   ├── model.safetensors      # The actual weights
  │   ├── tokenizer.json         # Tokenizer files
  │   └── config.json            # Model architecture (Layers, Heads)
  │
  ├── prompts/                   # VERSIONED GIT (Templates)
  │   ├── system_v2.txt          # "You are a helpful assistant..."
  │   └── tasks_v1.yaml          # Task-specific templates
  │
  └── config/                    # EXPLICIT (Runtime Settings)
      └── inference_prod.yaml    # Generation params (Temp, Max Tokens)

Why this structure matters:

  1. No Runtime Downloads: The /models folder must contain everything needed to run. If your code tries to download model.safetensors from Hugging Face when the server starts, your service will fail the moment the network blips.
  2. Independent Rollbacks: Two weeks from now, if a prompt change breaks quality, you can revert the /prompts file instantly without restarting the heavy model server.

4: Input/Output Safety (The Request Boundary)

Setup validations for inputs before they reach your model by rejecting bad requests early. You want to setup validations for all incoming requests to your LLM. It’s your first line of defense to help with latency (rejects expensive requests early), cost (prevents runaway generation) and output quality (enforces sane bounds). In production, that translates to:

  • Input side: Max input tokens (you can’t cook a meal that requires more burners than you have).
  • Output side: Max output tokens (portion control—for cost and timing).

Before vLLM sees any request, validate it:

python
def validate_request(prompt: str, max_tokens: int) -> None:

    # Check raw input length
    if len(prompt) > MAX_INPUT_CHARS:
        raise ValueError(f"Input too long: {len(prompt)} chars")

    # Rough token estimate (most tokenizers: ~4 chars per token)
    # This catches expensive requests before they hit the model
    estimated_tokens = len(prompt) // 4
    if estimated_tokens > 6000:  # Leave room for output in 8K context
        raise ValueError("Estimated input tokens exceed limit")

    # Enforce maximum output length to control costs and latency
    if max_tokens > 1024:
        raise ValueError(”max_tokens too high”)

    # Reject empty or malformed prompts
    if not prompt.strip():
        raise ValueError("Empty prompt")

This validation layer lives in your API wrapper, the code between users and vLLM. Every incoming request passes through this check first.

Note: Early on, it’s almost always safer to reject requests than to queue them. Queues hide overload until everything fails at once. Rejections fail fast and keep the system responsive for users who do get through. Returning a clear 429 (“Too Many Requests”) is better than letting latency balloon or crashing the process. You can add smarter queuing later but only after you understand your true capacity.

5: Observability (Logging & Monitoring)

You’re not logging to prove nothing went wrong. You’re logging so that when something does go wrong, you can figure out why. This is especially true for fine-tuned models. They are more sensitive than base models; a small change in a prompt or a slight shift in input context can cause quality to regress significantly. You need tight visibility.

The “Kitchen Fire” Manifest:
For fine-tuned models, a few signals carry most of the insight. If you track nothing else, track these:

  • Time-to-first-token (TTFT): What users feel in the wait for a response
  • P95 latency: Slowest 5% of requests where overload and batching issues surface first.
  • Error and refusal rates: Stability*.* Sudden jumps usually mean a prompt or load change broke something.
  • Tokens per request: Average input + output tokens per call define cost and capacity. As this creeps up, throughput drops and GPU costs explode.
  • GPU memory & restarts: Uptime health. Sustained redlining means instability, not efficiency.

The Trade-offs:

  • Privacy: You need to see what users are asking to debug quality issues, but you can’t violate privacy. Strategy: Hash sensitive identifiers or redact PII entities, but keep the structural context of the prompt.

  • Storage Costs (Sampling):

    • Early days (Low Traffic): Log nearly 100% of requests. You need the data to understand baseline behavior. Start simple by writing structured logs (JSON) to stdout.
    • At Scale (High Traffic): Log 1–10% of successful requests, but always log 100% of errors and latency outliers (e.g., requests taking >5s).
    • Your container or cloud provider (AWS CloudWatch, Datadog, etc.) will capture these automatically. Don’t build a complex logging pipeline until you actually need it.

6: Resource Management (Capacity)

Defaults are dangerous. You’d test it. Same here. Inference engines ship with safe, generic defaults. If you don’t tune them, your context window will grow unchecked and latency will degrade silently. Launch your server with explicit capacity flags.

python -m vllm.entrypoints.openai.api_server \
  --model /models/my-finetuned-v3 \
  --max-model-len 8192 \             # Hard cap on context
  --gpu-memory-utilization 0.90      # Reserve memory upfront

Flags That Actually Matter

This is where many production issues are quietly introduced. The “right” values depend on your specific model and hardware. Test under load before declaring victory and putting into production. Leaving these at defaults means vLLM is guessing. You should be choosing. In a restaurant, you wouldn’t guess how many burners you have or how many tables you can serve.

–max-model-len 8192 Caps context window. This is a hard limit. Requests beyond this get rejected, not silently truncated.

–gpu-memory-utilization 0.90 Controls how much GPU memory vLLM reserves. Too high = OOM crashes. Too low = wasted capacity. Granted 0.90 can be aggressive on some cards/models and 0.85 may be a better place to start. Start conservative.

Finding Capacity:
Your first goal isn’t maximizing throughput, it’s finding the cliff. Default to rejection over queuing until you’ve measured real capacity. Push concurrent requests until TTFT or error rates spike, then back off. That number is your real capacity. Capacity math doesn’t need to be perfect, but it does need to exist.

7: Deployment Strategy (Rollout)

Even a perfect deployment shouldn’t go from zero to full traffic instantly. The safest way to ship is to limit blast radius while you learn how the model behaves under real load. Core phases for rollout that are good to apply when you can:

  1. Phase 0 (Staging): Run load tests in staging. Never let production be the first time code hits a GPU.
  2. Phase 1 (Shadow): Send production traffic to both models. Return the old model’s response to the user. Log the new model’s performance silently (don’t share with the user) to check for errors/latency. This typically requires your API layer to duplicate requests to both endpoints, or using tools like Istio/Envoy that can mirror traffic. Observe latency and errors silently.
  3. Phase 2 (Canary): Route 1-5% of users to the new model and let them see the results. Watch for complaints or regression.
  4. Phase 3 (Cutover): Shift 100% traffic when stable.

Rollouts are about giving yourself time to notice problems while they’re still easy to undo.

8: Operational Resilience (When Things Break Which They Will)

In production, your model isn’t just a script running on your laptop; it’s a service that needs to survive crashes, traffic spikes, and recover gracefully. The core concepts to embrace here:

Health Checks: Most orchestration tools (like Kubernetes) ask two questions. Do not confuse them.

  • Liveness (”Are you there?”): If this fails, the system restarts. Use a simple ping for this.
  • Readiness (”Can you cook?”): If this fails, the system stops sending traffic but keep the model running. Use the following example code:
# BAD: Lazy check (Are we alive?)
@app.get("/health")
def health():
    return {"status": "ok"}

# GOOD: Readiness check (Can we actually work?)
@app.get("/ready")
def ready():
    try:
        # Actually try to generate a tiny token.
        # If this fails, we are not ready to receive user traffic.
        response = engine.generate(
            prompt="test", 
            sampling_params={"max_tokens": 1},
            timeout=5
        )
        return {"status": "ready"}
    except Exception as e:
        # Return 503 so the load balancer knows to wait
        return JSONResponse(status_code=503, content={"error": str(e)})
  • The Trap: If you use a “Liveness” check to see if the model is loaded, the system will see your model loading (which takes time), think it’s broken, and kill it before it ever finishes.
  • If you’re running without orchestration (just a VM), implement these as API endpoints: check /health/ready before sending traffic, and monitor /health/live with a cron job or external monitoring tool.

The “Cold Start” Reality: Large LLMs take 30–60 seconds to load into GPU memory.

  • During this window, your Readiness Probe is your shield. It prevents users from hitting the server until that 60-second process is complete.
  • If you don’t use readiness checks, the first 50 users after a deploy will see errors while the model loads.

Graceful Shutdowns: When you deploy a new version, the old one needs to die.

  • The Wrong Way: The server stops instantly. Anyone currently generating a response gets cut off mid-sentence.
  • The Right Way: The server stops accepting new requests but stays alive for 30–60 seconds to finish current generations. (Configurable in uvicorn and your orchestration timeouts).

Define Your Failure Paths Silent failures are worse than clear ones. You need to decide explicitly what happens when limits are hit:

  • On Timeout: If prep takes too long, do you return a partial response or a hard error? (Usually hard error).
  • On Memory OOM: When the kitchen is at capacity, do you queue the request or reject it? (Rejecting with a 429 “Too Many Requests” is often safer than crashing the queue).
  • Circuit Breakers: If 5 requests fail in a row, stop taking orders immediately. Give the system 30 seconds to recover before trying again.

Why this matters: If your service can’t fail safely, it will fail loudly. Better to tell a customer “we’re at capacity” than to leave them waiting indefinitely.

After Deployment: What Changes

Once your API is live, the controlled environment of the test kitchen vanishes. Customers will order differently than you expected. Inputs will drift. Traffic patterns will shift. The prompt that worked perfectly in your test suite will fail on a real user’s 5,000-word messy input.

This isn’t a deployment problem; it’s just reality. But the infrastructure choices you made determine whether you can see these changes and respond, or whether you’re scrambling in the dark. Most deployment disasters aren’t exotic. They happen when teams ignore this reality and fall into predictable traps.

What Actually Goes Wrong

  • The “Defaults” Trap: Ignoring Essential 6. You launch with vLLM’s defaults. Context grows unchecked until the server falls over.
  • Phantom Changes: Ignoring Essential 3. Prompts get tweaked “just to see.” Quality regresses, and nobody knows what changed.
  • Testing Only the Happy Path: Ignoring Essential 4. You tested short, clean prompts. Real users send malformed JSON. If you haven’t tested your failure logic, the first edge case will take down the service.
  • Flying Blind: Ignoring Essential 5. Logs are either non-existent or overflowing with noise. If you can’t trace a specific request_id from input to error, you can’t debug.

Good deployment practices don’t eliminate problems. They make problems solvable. Versioning lets you revert a bad prompt. Boundaries prevent one heavy request from crashing the server. Instrumentation tells you why latency spiked. Deployment discipline is simply the difference between a system that breaks chaotically and one that breaks manageably.

What “Done” Actually Looks Like

A restaurant isn’t ready for opening night just because you finished testing the recipes. It’s ready when the kitchen can handle service without you standing over every order.

You’re “done” when you stop guessing. When a prompt tweak tanks quality, you can roll it back without redeploying the world. You know which version is running and when traffic surges and the system alerts before it hits capacity and if it does then it handles it instead of faceplanting.

When you deploy, things will never go according to plan but don’t let that stop you from launching. Fine-tuning gets you the recipes you want. Deployment discipline is what lets you keep serving them on opening night, and the hundred nights after.