LLMs on Nyghtowl

Swapping Models in the Agent House

Sat, 25 Jul 2026 17:13:21 +0000

The grief for GPT-4o was surprising and fascinating when it was removed and retired and this is not isolated to that model. It’s something that has happened with other models and there is still some sadness on when they change but maybe we are at a stage where it’s less impactful and more understood (or we are just jaded). Someone else changes the weights, and users wake up to an AI that’s suddenly a different character.

Opening Night: Launching a Fine‑Tuned LLM to Production

Tue, 30 Dec 2025 20:53:57 +0000

Video & Podcast

Your fine‑tuned model is ready. Evals look good. Now you need to actually deploy it.

This is the part most guides skip: how do you take a checkpoint and turn it into a service that real users can hit? What infrastructure decisions matter? What boundaries do you set? What happens when the GPU runs out of memory on New Year’s Eve?

Think of it like opening a restaurant. Your recipes work great in the test kitchen. But opening night means: multiple orders arriving simultaneously, ingredients running low, equipment breaking, customers ordering things you never tested together, and kitchen staff who’ve never worked a real dinner rush.

Your Fine-Tuned LLM Model Isn’t Ready Yet: Here’s How to Evaluate It

Wed, 17 Dec 2025 16:57:45 +0000

Video & Podcast

Fine-tuning large language models has become dramatically easier. With techniques like QLoRA, teams can adapt billion‑parameter models on relatively modest hardware and get impressive results quickly. But this ease hides a trap: a model that finished training is not the same thing as a model that’s ready for production.

Many teams run a few spot checks, feel good about the outputs, deploy, and only discover weeks later that the model has drifted, slowed down, hallucinated, or lost user trust. The gap between “it trained successfully” and “it’s safe to ship” is wide, and closing it requires a testing mindset different from traditional software QA.

So You Want to Fine-Tune a Model (and You Finally Got a GPU!)

Thu, 11 Dec 2025 00:20:18 +0000

Podcast

You managed to get access to an H100 or A100 instance - great! Now comes the part no one really talks about: setting up the server so you can actually use it for fine-tuning. Cloud GPU setups have their quirks no matter where you run them, and having a clean, reliable configuration makes the difference between training tonight and debugging until sunrise.

If you’re spinning up an H100 or A100 VM for training, welcome = ) you’re about to do something fun, powerful, and occasionally puzzling. Every cloud provider has its own configuration rituals, and Azure is no exception. These notes capture what I wish Past Me had in front of her regarding the key steps.

How to Run Big Models on Small GPUs | All about Quantization

Wed, 26 Nov 2025 17:33:54 +0000

Video | Podcast

You’ve trained or fine-tuned a powerful LLM. It’s smart, it works, and it’s 51 GB. Now you need to deploy it on a 24GB GPU, or distribute it to users who don’t have data center hardware. Suddenly, that beautiful model doesn’t fit anywhere useful. This is the quantization problem, and it’s increasingly unavoidable.

Quantization is how we make big models fit on small GPUs. It’s like image compression for neural networks: a RAW photo might be 50MB, but compress it to JPEG and it’s 5MB. You lose detail, but to most people, it looks the same. A 50GB model becomes 13GB. A model that needed 80GB of VRAM (A100 territory) now runs on 24GB (like an RTX 3090 or 4090).

🧠 Building with LLMs: From Chat to Custom AI

Mon, 17 Nov 2025 17:58:54 +0000

Video & Podcast

Large Language Models (LLMs) can feel like magic to newcomers or maddeningly complex to veterans. Whether you’re automating workflows, building AI products, or just curious about the tech reshaping every industry, understanding how to use LLMs (and their multimodal cousins, MLLMs) is what we are here to discuss.

What’s an LLM, really? If you don’t know anything about LLMs, you can start by thinking of them as a really good at predicting language almost like autocomplete but that’s only a tiny piece of the picture. Realistically they’re much closer to a reasoning engine that’s trained on massive datasets that learns the structure of language and how ideas connect. An MLLM operates on multiple data types (thus multimodal): audio, video, images, and text. Example: upload a chart and ask for the trend.

LLM Fine-Tuning Optimization Part 2: Achieving Stability

Tue, 04 Nov 2025 15:13:38 +0000

Video & Podcast

In the LLM Fine-Tuning and the Performance Tug-of-War post, we explored the constant balancing act between quality, speed, and memory when fine-tuning large language models that align with your goals and constraints. Fine-tuning is the process of teaching an existing model (e.g. GPT-NeoX, Llama, Gemma, etc.) new knowledge or behaviors, transforming a general-purpose LLM into one that is a domain specialist.

The previous post introduced the core levers of sequence length, optimizers, attention type, and LoRA, which define how a model learns, generalizes, and scales. This post expands on your optimization toolkit by introducing four more essential levers: batch size, mixed precision, quantization, and gradient checkpointing.

When AI Models Start Talking to Each Other

Mon, 27 Oct 2025 16:06:53 +0000

For a while now, I’ve been using LLMs in a way that feels a bit like hosting a party for AIs. What happens when you don’t just prompt a single model but let multiple models talk to each other? To explore that question, and being inspired by the holiday, I built two demos that make the idea both technical and fun:

🧛 Monster Mash Chatroom: A FastAPI app where different LLMs wear costumes as classic Halloween monsters, chatting in real time.

What Is LLM Fine-Tuning?

Fri, 24 Oct 2025 16:07:25 +0000

Podcast & Video

When I wrote the LLM Fine Tuning & Performance Post, a good question that came up was: what exactly is fine-tuning and how does it work? This post can’t fully cover the topic, but I want to give some solid context around how fine tuning works for LLMs and MLLMs (the billion-parameter models that come from Anthropic, OpenAI, Mistral, Meta, DeepSeek, etc.).

Large language models are incredibly capable. They can write essays, debug code, or summarize medical records. But out of the box, they’re trained to handle everything in a general way, and they have limits based on the data they have access to. Granted, they continue to grow and improve in their capability and breadth of expertise so this is continually changing.

LLM Fine-Tuning & Performance Tug-of-War

Tue, 14 Oct 2025 18:40:03 +0000

Podcast

This year has been a crash course in large language model (LLM) development, marked by long training runs, GPU juggling, and a constant cycle of trial and error. While there is a lot of excitement around building custom LLMs, you probably know you shouldn’t start from scratch. Fine-tuning is the process of teaching an existing model (e.g. GPT-NeoX, Llama, Gemma, etc.) new knowledge or behaviors which transforms a general-purpose LLM into one that is a domain specialist. It bridges the gap between broad capability and domain-specific precision, allowing teams to achieve superior results without the immense cost of training from scratch.