Podcast

You managed to get access to an H100 or A100 instance - great! Now comes the part no one really talks about: setting up the server so you can actually use it for fine-tuning. Cloud GPU setups have their quirks no matter where you run them, and having a clean, reliable configuration makes the difference between training tonight and debugging until sunrise.

If you’re spinning up an H100 or A100 VM for training, welcome = ) you’re about to do something fun, powerful, and occasionally puzzling. Every cloud provider has its own configuration rituals, and Azure is no exception. These notes capture what I wish Past Me had in front of her regarding the key steps.

A note on versions: Throughout this guide, you’ll see placeholders for drivers, CUDA toolkits, and container images. These components update frequently, and what’s current today may be outdated in weeks. Always check the latest stable versions recommended for your specific workload, GPU model, and training framework the day you set up your instance.

What We’ll Cover

This guide walks you through the complete VM setup process for fine-tuning on H100 or A100 GPUs. These aren’t just configuration steps - they’re the decisions and details that determine whether you’re training tonight or debugging until sunrise.

Setup tips include:

  • Building the VM (OS selection, GPU choice, storage sizing, Secure Boot)
  • SSH and networking (secure access, private connectivity)
  • NVIDIA drivers (GPU stack setup and verification)
  • Hugging Face authentication (model access and credentials)
  • Git LFS (getting real model weights, not pointer files)
  • Screen sessions (keeping training alive through connection drops)

Getting these setup right will get you much closer to a reliable environment ready for fine-tuning. We’ll also cover a pre-training checklist and notes on post-training validation.

With that, let’s get into the field notes of the things that quietly matter for a smooth fine-tuning workflow.

1: VM Build (Where the Real Story Begins)

A few things matter more than the defaults, especially when you first configure your VM. This is the moment to set up the fundamentals like choosing the OS, GPU, adding your SSH public key, sizing storage, and disabling Secure Boot.

Pick your operating system.

Ubuntu LTS (22.04 or 24.04) is the safe default for GPU workloads. It has excellent NVIDIA driver support, extensive documentation, and most ML frameworks test against it first. If you have specific requirements or organizational standards that dictate a different distribution (RHEL, CentOS Stream, Debian), those work fine too - just be prepared to translate package manager commands (apt → dnf/yum) and verify NVIDIA driver compatibility for your chosen OS version.

Unless you have a compelling reason otherwise, stick with Ubuntu LTS. It removes a category of potential friction.

Choose your GPU with intention.

Availability varies by region (true everywhere, not just Azure), so check your quota before you pick your VM type. Nothing kills momentum like getting to the last screen only to learn your quota has strong feelings about your choices.

H100s and A100s are in high demand, so don’t be surprised if your initial quota request gets denied or delayed. It’s common to need to justify your use case, try multiple regions, or wait for capacity to open up. Plan ahead because quota approval can take days or weeks depending on the provider and region.

Add your SSH public key.

This must be added during VM creation so it gets installed on the machine. Cloud providers won’t automatically use your local key, so make sure it’s provided here. You must provide your public key (id_ed25519.pub or similar) in the VM setup screen so it gets placed into ~/.ssh/authorized_keys on the instance. Once that key is registered, you can connect normally.

Give yourself real storage.

Start with at least 1TB, but assess the size of your dataset, model, checkpoints, and any other artifacts you plan to keep on the machine. Some workflows are perfectly comfortable at 1TB. Others, especially those involving large datasets or multiple experiment runs, may need 1.5TB, 2TB, or more.

Right-size your storage up front so you’re not scrambling to free space mid-training.

Important: Turn off Secure Boot for H100/A100.

Secure Boot validates that kernel modules are signed by trusted keys before loading them. NVIDIA’s proprietary drivers aren’t signed with the keys that most Linux distributions trust by default, so Secure Boot will block them from loading - not because the drivers are unsafe, but because they’re not in the pre-approved trust chain.

Azure Portal → VM → Settings → Security → Secure Boot → Off.

You can manually enroll NVIDIA’s signing key to keep Secure Boot enabled, but most GPU training workflows simply disable Secure Boot since it’s faster and has no practical security impact for dedicated training VMs. Do this during initial setup to avoid the “why won’t nvidia-smi talk to me” detective arc later.

2: SSH + Networking (Start secure, stay secure)

Next up is setting up access to the VM and connecting. With the key you included in the configuration, you can ssh into the machine. SSH in with agent forwarding so your GitHub setup just works:

ssh -A -i ~/.ssh/id_ed25519 <username>@<public_ip>

Lock down network access

Your VM starts with a public IP and open SSH port which is convenient for initial setup, but not ideal long-term. The goal here is to establish private, secure access to your VM so you can close the public SSH port entirely.

Option 1: VPN or private network (recommended) Use a VPN solution like Tailscale, Wireguard, or your cloud provider’s native VPN service to create a private network connection. This lets you access your VM from anywhere without exposing SSH publicly.

For example, with Tailscale:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

After authorizing in the browser, you can connect using your VM’s Tailscale IP from any device on your network.

Option 2: Bastion host or SSH tunneling If your organization uses a bastion host, configure your SSH access through that. This keeps your training VMs on private networks only.

Once you have private access working, go back to your cloud provider’s networking settings and remove or restrict the public SSH rule. From here on out, you’ll connect privately - cleaner security posture, predictable connectivity, and no public attack surface.

3: NVIDIA Drivers (The Classic GPU Ritual)

Once inside the VM, get your GPU stack working.

Note: These instructions assume Ubuntu (which is widely supported and well-documented for GPU workloads). If you’re using a different Linux distribution, adjust package manager commands accordingly - dnf or yum for RHEL/CentOS/Fedora, zypper for SUSE, etc.

Start with essential tools:

sudo apt update && sudo apt install -y python3-venv python3-dev nvtop

Then install the NVIDIA drivers:

sudo apt install -y nvidia-driver-<version> nvidia-utils-<version>
sudo reboot

After reboot, verify your GPUs:

nvidia-smi

If you see all your GPUs, you’re good. If not, double-check Secure Boot. It’s almost always the culprit.

Pro tips:

  • Use nvtop as your dashboard during long runs. It shows real-time GPU memory, utilization, and thermal behavior - making it much easier to catch bottlenecks, detect stalls, or spot misbehaving processes before they derail your training.
  • H100s work best with CUDA 12.1+. Some training packages require specific CUDA versions, and your system CUDA may not match what these libraries expect. Always confirm compatibility when installing or upgrading packages.
  • Isolate your training environment using a Python virtual environment or conda. This prevents CUDA version conflicts between system libraries and training packages, keeps system CUDA untouched, and gives you a safe place to install optimized libraries without breaking anything.

4: Base Model Auth (Accessing the Model)

Before anything else, think about how you will access the base model you plan to fine tune. Many models require authentication - whether they are gated on Hugging Face or hosted in a private or organization-controlled repository. Authentication is the part that catches people off guard, so set it up early.

Start by installing the CLI:

sudo apt install -y python3-pip
python3 -m pip install --upgrade pip
python3 -m pip install “huggingface_hub[cli]”

Then log in:

hf auth login

If the command looks missing, add this to your PATH:

export PATH=”$HOME/.local/bin:$PATH”
echo ‘export PATH=”$HOME/.local/bin:$PATH”’ >> ~/.bashrc && source ~/.bashrc

Side note: The 401 gotcha

Sometimes you have all the permissions, and hf auth whoami even shows the correct account, but downloads still 401. This happens because some tools want the token exported explicitly.

export HF_TOKEN=”$(cat ~/.cache/huggingface/token 2>/dev/null || cat ~/.huggingface/token)”

It’s a quirk of environment variables, and this fixes it.

5: Git LFS (Avoid the Pointer File Heartbreak)

Before cloning any model repos:

sudo apt install -y git-lfs
git lfs install

This step is all about making sure you actually get the real model weights. Git LFS handles large files, and without it, you’ll only download placeholder pointer files instead of the actual tensors your training script needs. You need the real weight files because fine-tuning updates the underlying tensors. Without them, the model cannot load and training cannot start.

Example: What you get without Git LFS vs with it

Pointer file (without LFS):

version https://git-lfs.github.com/spec/v1
oid sha256:8f3c...c2a7
size 13421772800

This is only a reference - not the actual model.

Real weight shard (with LFS):

$ ls -lh
model-00001-of-00005.safetensors   3.2G
model-00002-of-00005.safetensors   3.2G
...

These are the actual tensors your training code needs.

Does this apply only to Hugging Face?

No. This applies to any base model stored in a Git repo using Git LFS for large files - including private repos, organization-managed repos, or self-hosted model registries. If the model weights are tracked with LFS, you must install Git LFS to pull the real files.

Where you’re likely to see LFS used

Git LFS is extremely common for:

  • Large language models with multi-shard .safetensors files
  • Vision and multimodal models with multi-GB backbones
  • Repositories that include pretrained checkpoints or optimizer states
  • Research repos where large datasets or training artifacts are checked in

Any time a repo stores files larger than the standard Git size limits, LFS is typically involved.

6: Screen Sessions (Your Lifeline)

This step is about keeping your training process alive even when your SSH connection is not. Fine-tuning jobs can run for hours or days, and network interruptions are inevitable. screen ensures your training continues safely in the background so you don’t lose progress.

Use screen:

screen -S training

Detach with Ctrl + A, then D. Reattach with:

screen -r training

This tool saves more sanity than coffee .

Quick Pre-Training Checklist

Before you kick off a long run, verify the essentials:

Access & connectivity:

  • Private network access is working (VPN connected, or bastion accessible)
  • You’re inside a screen session

GPU stack:

  • nvidia-smi lists all GPUs with the correct driver
  • nvtop is running and displaying GPU activity

Code & data:

  • Repository is cloned and on the correct branch
  • Dataset is copied to the VM
  • Git LFS is installed and pulling real weights, not pointers

Authentication:

  • Hugging Face authentication works (hf auth whoami succeeds)
  • Any other model/data repos are accessible

Final Thoughts

Setting up an H100/A100 VM isn’t hard. It’s just a sequence. Once you learn it, everything clicks. GPU VMs everywhere have their “first-time surprises,” but after one or two setups, it becomes second nature.

The checklist matters: Secure Boot off, drivers verified, auth sorted, screen running, and most importantly, run a short test first. Running a 5-10 minute training pass on a tiny dataset before committing to the real run catches misconfigurations you don’t want to discover 18 hours deep.

Automating these steps is also worth the effort. They translate well across cloud providers, and scripting them removes a whole category of setup friction. There are also increasingly powerful self-serve fine-tuning platforms that abstract away much of this VM setup entirely which let you fine-tune models without configuring drivers, CUDA, networking, or storage. They can be great options when you want to focus purely on the fine-tuning workflow rather than infrastructure, but understanding the underlying setup gives you far more control and helps you troubleshoot when things get tricky.

A note on testing and validation: Once your fine-tuning completes, you’ll want to validate and test the resulting model. Testing is its own deep topic and many workflows use containerized inference frameworks like VLLM in Docker with GPU support to create isolated, production-like environments for evaluation. That’s a separate setup you can tackle after your first successful training run, but it’s worth keeping in mind as you think about your full workflow.

Happy training.