Hot-Water Cooling in HPC: How Data Centres Can Use Energy More Efficiently

An HPC system needs electricity for more than computation alone. Pumps, fans and, where required, chillers also consume energy to remove the heat it produces. Direct hot-water cooling addresses this additional demand. It can also help make waste heat available for use outside the data centre. Both benefits are valuable, but they should be assessed separately.

How can hot water cool a computer?

The term may sound contradictory at first, but it does not mean boiling water. The coolant must be sufficiently cooler than the components being cooled, while keeping them within their permitted operating temperatures. It can then absorb heat even if it already feels warm or hot to the touch.

In direct liquid cooling, the coolant flows through components such as cold plates attached to processors. It absorbs heat close to where it is generated and carries it away through a cooling circuit. A US Department of Energy report on HPC cooling in Maui describes this principle using direct cooling of processors and memory, with heat rejected to the outside air.

The permitted supply temperature depends on the server and the overall cooling design. Warm-water or hot-water cooling is therefore not a blanket approval for any particular temperature: the specifications of the individual system remain decisive.

Where do the efficiency benefits come from?

Less energy needed to generate chilled water

The higher the permitted cooling-water temperature, the more often heat may be released to the outside air without energy-intensive refrigeration. This principle is known as free cooling. Whether it works throughout the year depends on factors including the local climate, temperature limits and the design of the heat-rejection equipment.

Free cooling still requires electricity: pumps and, where necessary, fans remain in operation. The Leibniz Supercomputing Centre uses SuperMUC-NG as an example to explain how warm-water cooling can substantially reduce the need for additional cooling equipment (article in German).

Less heat to remove through the room air

If a large share of the heat is transferred directly into the liquid circuit, less needs to be removed through the room air. The actual cooling coverage is crucial: a solution for CPUs and GPUs does not automatically cover power supplies, memory and networking components. Any remaining air cooling must be included in the design.

Reusing waste heat: An additional benefit, not automatic energy savings

Warm return water can serve as a heat source for buildings or heating networks, for example. For this to work, the temperature level, distance and timing of heat demand must align. If the recipient needs higher temperatures, a heat pump may be required. Its energy consumption must also be included in the assessment.

CAPELLA at TU Dresden provides a concrete example. The MEGWARE case study on CAPELLA (in German) describes servers cooled directly with water at 35 °C, without server fans. According to the case study, more than 95 per cent of the waste heat is removed via warm water, while heat pumps support its use in the district heating network.

This figure refers to the share of heat removed through water, not electricity savings of 95 per cent. It is also specific to this project and cannot be applied universally to other HPC installations. Sound planning must also establish where the heat will go when there is no demand for it.

How can energy efficiency be assessed?

PUE describes infrastructure overhead

Power Usage Effectiveness, or PUE, is the ratio of a data centre's total energy consumption to the energy used by its IT equipment. Both figures must use the same measurement boundary and time period. The US Department of Energy's explanation of PUE uses annual energy figures for this calculation.

A purely illustrative calculation: if IT consumes 1,000 MWh and total consumption is 1,200 MWh, the PUE is 1.20. The additional 200 MWh is used by supporting infrastructure, not exclusively by cooling. If total consumption falls to 1,100 MWh while IT consumption remains unchanged, the PUE is 1.10. This example is not a forecast for any particular system.

The computing work completed still matters

A low PUE alone does not tell you how much energy a simulation needs to reach a result. Operators should therefore also examine the runtime and energy consumption of representative computing jobs. Utilisation and software optimisation can also affect the overall energy balance. Usable waste heat, water consumption and energy sources add to the assessment; PUE alone does not describe them.

What should be clarified before implementation?

The following checklist can help with project preparation:

• IT and temperatures: Which components will be cooled directly, and what supply and return temperatures are planned?
• Site and load: How will cooling work in summer heat, at partial load and after future expansion?
• Operation: How will water quality, leak detection, maintenance and resilience be managed?
• Heat reuse: Who will use the heat, when, and at what temperature level?
• Verification: Which measurement points and comparison conditions will make energy consumption and costs transparent?

Conclusion: Plan cooling and computing performance together

Hot-water cooling can reduce the additional energy demand of an HPC data centre and make useful waste heat reuse easier. The scale of the benefit, however, depends on the specific system as a whole. A convincing solution combines suitable IT with site-appropriate cooling technology and an operating concept that can be verified through measurement.

Would you like to consider energy efficiency from the outset in your next HPC project? Find out more about our HPC solutions and discuss your requirements with MEGWARE.