Data Center Liquid Cooling Training: What Technicians Must Learn for AI Racks

Why liquid cooling changes the technician's job

In an air-cooled hall, a team usually has minutes to respond to a cooling problem. Direct-to-chip cooling leaves far less time. The coolant runs through cold plates on the GPUs and CPUs, and if the flow stops, the chips heat up fast.

An ASHRAE TC 9.9 technical bulletin on cold plate resiliency cites research with medium-power processors in which throttling began as little as 23 seconds after a CDU pump failure, and today's AI processors run at higher power. The bulletin also reports TCS temperature jumps of as much as 27°C (48.6°F) during a CDU pump changeover on systems without active redundancy.

Data center liquid cooling training has to teach technicians to recognize a fault, choose the correct response, and carry out the steps in the correct order, even though few of them have ever seen a real cooling fault.

How direct-to-chip cooling works: FWS, TCS, and the CDU

Most direct-to-chip systems use two separate liquid loops with a CDU between them, as the ASHRAE TC 9.9 white paper on water-cooled servers describes.

• The technology cooling system (TCS) is the closed loop that runs from the CDU to the racks, through manifolds, hoses, quick disconnects, and cold plates.
• The facility water system (FWS) is the building loop. It carries heat from the CDU to chillers, dry coolers, or cooling towers.
• Between them sits the coolant distribution unit (CDU), a liquid-to-liquid heat exchanger with its own pumps, filters, valves, and controls.

According to the white paper, the most common water quality mistake is to apply the wrong guideline to the wrong loop, and TCS water quality needs a clear owner. That makes the loop boundaries the first thing to teach.

What technicians must learn about the CDU

The ASHRAE white paper lists the main jobs of the CDU: it transfers heat from the TCS to the FWS, circulates the TCS coolant, keeps the TCS coolant chemistry separate from the facility loop, and holds the TCS above the room dew point to prevent condensation.

Know the normal state

To spot an abnormal reading, a technician first needs a clear picture of what normal looks like. For each CDU, trainees should be able to state:

• Which pump is lead and which is standby.
• The supply temperature set point and the normal return temperature.
• Normal TCS flow and differential pressure, and their alarm limits.
• Normal filter differential pressure, and the value that calls for a filter change.
• The reservoir level and the normal position of each isolation valve.

They should also be able to read active alarms and alarm history on the CDU screens, and know which actions need a method of procedure (MOP).

Quick disconnects and rack connections

Quick disconnects (QDs) let a technician connect a rack to the TCS without draining the loop. They are also a common spot for slow weeps, so follow the manufacturer's procedure every time.

Rack connection checklist

1. Inspect both halves for damage, debris, and seal condition. Keep the dust caps on until connection.
2. Verify supply-to-supply and return-to-return before you energize anything.
3. Engage the coupling fully, and confirm that it is latched.
4. Bleed air at the vents after each rack or bank of racks.
5. Look for drips under flow, and again at operating temperature.
6. Record the connection, the readings, and any fluid makeup.

Steps 2 and 4 come from the ASHRAE TC 9.9 bulletin on TCS commissioning. The same bulletin warns that the factory fluid in new IT equipment can differ from the site fluid, and that incompatible fluids can cause inhibitor precipitation, corrosion, or fouling. Check compatibility before the first connection.

Leak detection and leak response

The Open Compute Project white paper on leak detection and intervention divides detection into two groups:

• Direct detection uses leak detection rope, point detectors, and floats. The liquid has to touch the sensor and must be electrically conductive.
• Indirect detection relies on changes in differential pressure, a falling reservoir level, optical sensors, and turbidity sensors.

The paper also sets out levels of intervention. With manual intervention, the system sends a notification and people act on it. Automatic intervention can de-energize the IT equipment and, in some designs, shut off the coolant as well.

A leak response drill

1. Read the alarm: which sensor, which zone, and which CDU.
2. Check the CDU for a falling reservoir level or a change in pressure.
3. Go to the location with the correct PPE. Do not touch energized equipment in standing liquid.
4. Find the source, and use the site's emergency operating procedure (EOP) to decide what to isolate.
5. Tell the IT owner before any action that stops cooling to IT equipment.
6. Clean up, record the fluid lost and the makeup added, and open a work order.

Pump failover and loss of cooling

The ASHRAE resiliency bulletin recommends active redundancy on the TCS and FWS loop pumps, UPS power so that the TCS keeps running when utility power fails, and thermal energy storage to carry cooling through the transfer to generator power.

Technicians need to know how their own site's design behaves. During a pump trip, they work through these checks:

Filtration and fluid quality

Cold plates have very narrow channels, and particles, corrosion products, or biological growth can block them.

The Open Compute Project guidelines for propylene glycol-based heat transfer fluids describe two filtration methods. Sidestream filtration continuously sends a small share of the flow, typically 5 to 15 percent, through a filter, so the full volume passes the filter at least once a day. Full-flow inline filtration treats all of the flow. Many CDUs also carry coarser inline filters, such as 25 or 50 micron, to protect the heat exchangers and quick disconnects.

The ASHRAE TCS bulletin lists the minimum fluid parameters to measure: pH, conductivity, glycol concentration, inhibitor reserve, and turbidity. It also says to take samples only after the system reaches thermal and hydraulic steady state, since a sample taken during a transient will not represent the loop.

Fluid mistakes to train out

• Topping up with the wrong fluid or the wrong concentration.
• Filling glycol too fast, which causes foam and trapped air.
• Ignoring a slow rise in filter differential pressure.

What the ASHRAE and OCP guidance covers

Start with the IT equipment manufacturer's requirements. The ASHRAE resiliency bulletin says that the manufacturer's coolant quality specifications must be followed.

ASHRAE uses S-classes to classify the maximum coolant inlet temperature to IT equipment. When the resiliency bulletin was published, they ranged from 30°C to 50°C (86°F to 122°F). Our guide to data center commissioning levels shows where liquid cooling fits into the testing stages of a new build.

A data center liquid cooling training curriculum

The modules below are a practical starting point. Adapt them to each site's equipment and procedures.

Assess each module with a practical exercise in addition to any quiz. Our article on data center technician skills assessment explains how to set one up, and data center training programs covers the structure of a full program.

Practice faults before live work

Real faults are rare, and nobody should trip a pump on a production AI cluster to train a new hire. A good simulated drill starts from the normal state, introduces one fault, and asks the trainee to find it and respond. It records each action and the order it happened in, then tells the trainee specifically what they missed and why it matters.

Fault drills to include

• The CDU lead pump trips, and the standby pump starts.
• Filter differential pressure rises above the alarm limit.
• A leak alarm in one rack zone, with a falling reservoir level.
• TCS supply temperature rises because the FWS supply is too warm.
• Air in the loop after a rack connection, with unstable flow.

After the simulation, move trainees to supervised live work on low-risk tasks. Add these drills to data center technician onboarding and repeat them on a schedule. A browser-based simulation runs on a supported computer with a modern browser, and our VR vs. browser simulation training article compares it with headset-based training.

How Oasis helps

We build browser-based 3D simulations of data center equipment and procedures. Each one follows the customer's equipment, procedures, and site standards, and we review the steps and feedback with the customer's team.

For liquid cooling, we can simulate your own CDUs, manifolds, quick disconnects, and racks, along with the faults your team has to handle, such as a pump trip or a leak alarm. The CDU and GB200 rack scenes on this site are examples, and we build to whatever your site needs. This work supports our onboarding, training, and operations and maintenance data center programs.

You can try a data center simulation in your browser now. To talk about liquid cooling training for your site, book a 15-minute call.

Frequently asked questions