GPUs on Nyghtowl

So You Want to Fine-Tune a Model (and You Finally Got a GPU!)

Thu, 11 Dec 2025 00:20:18 +0000

You managed to get access to an H100 or A100 instance - great! Now comes the part no one really talks about: setting up the server so you can actually use it for fine-tuning. Cloud GPU setups have their quirks no matter where you run them, and having a clean, reliable configuration makes the difference between training tonight and debugging until sunrise.

If you’re spinning up an H100 or A100 VM for training, welcome = ) you’re about to do something fun, powerful, and occasionally puzzling. Every cloud provider has its own configuration rituals, and Azure is no exception. These notes capture what I wish Past Me had in front of her regarding the key steps.

How to Run Big Models on Small GPUs | All about Quantization

Wed, 26 Nov 2025 17:33:54 +0000

Video | Podcast

You’ve trained or fine-tuned a powerful LLM. It’s smart, it works, and it’s 51 GB. Now you need to deploy it on a 24GB GPU, or distribute it to users who don’t have data center hardware. Suddenly, that beautiful model doesn’t fit anywhere useful. This is the quantization problem, and it’s increasingly unavoidable.

Quantization is how we make big models fit on small GPUs. It’s like image compression for neural networks: a RAW photo might be 50MB, but compress it to JPEG and it’s 5MB. You lose detail, but to most people, it looks the same. A 50GB model becomes 13GB. A model that needed 80GB of VRAM (A100 territory) now runs on 24GB (like an RTX 3090 or 4090).