Optimize thermals on inference

#Why thermals matter for an inference server

A GPU protects its chip: when the temperature or power exceeds what its firmware allows, it lowers its clock speeds. This mechanism is called throttling. The nvidia-smi documentation distinguishes its causes: a software temperature-related slowdown (“sw thermal slowdown”), a more severe hardware slowdown that reduces clock speeds by a factor of 2 or more (“hw thermal slowdown”), and a power limit (“sw power cap”). If all these reasons are shown as inactive, the clock speeds run as high as possible.

The threshold isn’t the same on every card. NVIDIA defines it as “GPU Slowdown Temp,” the temperature at which the hardware starts optimizing clock speeds to cool down, and exposes it in nvidia-smi. So don’t rely on a general figure like “83 °C”: read your card’s value. Throttling causes lower speeds and, if the fans compensate, more noise; over the long term, a card that runs hot all the time ages faster, but we don’t have a reliable figure to cite on this point.

#Measure before tuning: what the card really tells you

The starting point is determining whether the card is throttling and why. Two commands are enough on a NVIDIA card. The first continuously displays the temperature, power, core frequency, and utilization; the second details the throttling reasons and the temperature thresholds for your model.

nvidia-smi --query-gpu=timestamp,temperature.gpu,power.draw,clocks.current.graphics,clocks.current.memory,utilization.gpu --format=csv -l 1

nvidia-smi -q -d PERFORMANCE
nvidia-smi -q -d TEMPERATURE

rocm-smi --showtemp --showpower --showuse

Run these commands during a generation lasting several minutes, not while idle. Two readings matter. If the core frequency drops while the temperature rises and a thermal reason is active, it's a cooling problem. If the core stays high but a power limit is active, the card is limited by its power consumption: lowering the limit further won't have a noticeable effect on generation speed, but it will reduce heat.

#The power limit: the setting with the biggest payoff

Capping power is simpler than adjusting the voltage curve, and reversible. For memory-bound inference, the loss is small up to a certain threshold, then becomes abrupt. The runaihome blog, which measured a RTX 3090 with a dense 27-billion-parameter model, reports that from 350 W to 250 W the card loses less than 3% of its speed for 100 W less power, and that at 200 W the core frequency collapses and speed drops by 35%.

For a RTX 4090, the same blog cites Tom's Hardware, according to which limiting power to 70% (about 315 W) retains about 94% of performance. These figures come from secondary sources and a specific card: do not apply them as-is to yours. The reliable method is to measure your throughput during a long generation, lowering the limit in 25- to 50-W increments and stopping before the drop.

# Lire la limite par défaut, minimale et maximale
nvidia-smi -q -d POWER

# Appliquer 250 W sur la carte 0 (nécessite les droits root)
sudo nvidia-smi -i 0 -pl 250

The nvidia-smi documentation specifies that the value must be between the minimum and maximum limits reported by the card, and that the command requires root privileges. After a reboot, check that the limit is still applied: if it has reverted to the default value, create a systemd unit to reapply it at startup.

[Unit]
Description=Limite de puissance GPU
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -i 0 -pl 250
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable it with sudo systemctl enable --now nvidia-power-limit.service.

#Fan curve: ramp up early rather than abruptly late

The factory curve prioritizes silence: slow fans up to a high temperature, then a ramp. For a sustained load, a curve that starts earlier keeps the card cooler at comparable noise levels because it avoids speed spikes. On Windows, MSI Afterburner adjusts the curve; on Linux, LACT is a graphical AMD, NVIDIA, and Intel GPU control application that includes fan control.

These values are a starting point, not a manufacturer recommendation. Set the final step based on your card’s “GPU Slowdown Temp” value, and judge the result by the noise you can tolerate in the room. On macOS, Apple manages the fans: there is no curve adjustment.

#Undervolting: finer-tuned, longer to validate

Undervolting means achieving the same clock speed with lower voltage, and therefore less heat. In Windows, configure it in MSI Afterburner (the voltage-frequency curve); in Linux, the common approach is to lock frequencies with the nvidia-smi option that sets the core frequency range. The benefit depends on the card: every chip has its own headroom, and an overly aggressive setting causes errors or crashes later, not immediately.

1. 01
   Start with a power limit that works

   Start with the power limit: it's the safest gain. Add an undervolt only if the card remains too hot or too noisy.
2. 02
   Set a maximum frequency

   Lock the core frequency below the maximum boost frequency, for example with nvidia-smi -lgc followed by a frequency range in MHz.
3. 03
   Test for a long time

   Run continuous generation for at least 30 minutes. Instability appears as CUDA errors, driver crashes, or incoherent output.
4. 04
   Compare

   Measure speed and temperature before and after on the same model and prompt, then keep the setting only if the speed loss is negligible.
5. 05
   Cancel if needed

   Reset the frequencies (nvidia-smi -rgc) or restart to restore the factory settings.

#Case and airflow: the improvement is measured in degrees

Airflow path

A clear front-to-back airflow path, with fresh air directed toward the card's fans. Cabling that obstructs airflow costs you degrees.

Filters and dust

A clogged filter reduces airflow: clean it regularly, and blow dust out of the card's heatsink with compressed air, keeping the fans from spinning so they do not overspeed.

Clearance around the card

Two cards mounted tightly together in a multi-GPU setup heat up the first one; space them apart or add a side fan.

Environment

A room at 30 °C versus 20 °C adds those ten degrees to the card: a server in a closed closet is a thermal problem before it is a card problem.

To evaluate a setting accurately, keep the protocol constant: use the same model, the same long prompt, and the same generation time, changing only one parameter at a time. Record the peak temperature, core frequency, and average throughput at the end of generation, not the beginning, because a cool card always produces better numbers than the same card after twenty minutes. Repeat the measurement at the same ambient temperature; otherwise, the comparison is skewed by the room, not the setting.

#Repasting: a last resort, with its risks

After several years, the thermal paste and pads on a used card may have dried out. The symptom is a higher temperature than when you bought it under the same load, despite a clean case. Taking apart a graphics card voids the warranty, can damage the pads and chip, and requires care. Only do it after ruling out simpler causes: dust, airflow, power limit, and ambient temperature. If you're not sure, take it to a service provider; compare the cost with that of a replacement card.

#Laptops and Macs: the levers are different

On a laptop, you can’t adjust the voltage or thermal paste. The useful lever is the power mode. Apple describes two modes: Low Power mode, which, on macOS Sequoia 15.1 and later, also reduces fan noise on models with active cooling, and High Power mode, which lets the fans spin faster to maintain performance under very intensive workloads, with additional noise. For a fanless MacBook Air, the only levers are a heat-dissipating stand, AC power, and a smaller model.

• macOS settings for Apple Silicon
• MacBook Pro M2 Pro / Max
• MacBook Air M3

• Choose your GPU for local AI
• 32 GB AI PC build
• Multi-GPU setup with llama.cpp
• Source: nvidia-smi documentation
• Source: power-limit measurements (runaihome)
• Source: Apple, Mac power modes

#Frequently asked questions