01 Operational parameters & standard limits
Target operating windows for a direct-to-chip CDU secondary (technology) loop. Use these as the acceptance band during commissioning and the alarm band during operation. Site values may differ per vendor data sheet and chip TDP — always confirm against the unit's manual and the IT hardware spec.
| Parameter | Typical target / limit | Notes & standard |
|---|---|---|
| Secondary supply temp (TCS supply) | ~17–45 °C | ASHRAE TC 9.9 W17–W45 classes; lower for high-TDP GPUs. Number = max supply °C. |
| Secondary ΔT (supply→return) | ~8–12 °C | Sized with flow to the rack heat load (Q = 4.186 × L/s × ΔT). |
| Coolant flow rate | ~1.0–1.5 LPM / kW | OCP guidance ~1.5 LPM/kW at a 10 °C rise; per-cold-plate min flow per chip vendor. |
| Differential pressure (CDU/manifold) | ~0.5–3.0 bar | Within the cold-plate's min/max dP window; watch for filter-clog rise. |
| System (loop) pressure | ~2–6 bar | Below component PN rating; maintain with expansion/make-up control. |
| Coolant / fluid | 25–30% PG (or DI / dielectric) | Propylene-glycol mix for freeze/biocide; DI water (ASTM Type II) for some TCS; dielectric for immersion. Per vendor wetted-material list. |
| Water quality | ASHRAE W1–W4 | Conductivity, pH, hardness, particulate per ASHRAE liquid-cooling guidelines. |
| Filtration rating | ~50 micron (integrated) | e.g. Vertiv CoolChip integrated 50-micron filter; supplement with side-stream if specified. |
| Pump redundancy | N+1 | Dual/redundant pumps + dual power feeds (e.g. CoolIT CHx N+1 pumps + N+1 PSUs). |
| Leak detection | 0 leaks · rope/zone | Leak-detection rope/sensors at CDU base, manifolds, QDs; alarm → isolate. |
| Approach temp (ATD, L2L) | ~3–5 °C | Heat-exchanger approach (e.g. Boyd 3 °C, CoolIT 5 °C ATD). Higher ATD = fouling. |
| Supply vs room dew point | ≥ +2–3 °C above dew point | Dew-point-aware reset prevents condensation on plates/pipe. Critical safety interlock. |
| Pipe velocity (sizing) | ~1.5–3 m/s | Too slow → fouling/air; too fast → erosion + high dP. Sets pipe DN for the flow. |
| Corrosion inhibitor reserve | > 100 ppm azole | No nitrites; depletes over time. Cu < 0.2 ppm / Fe < 0.1 ppm = corrosion indicators. |
| Pump NPSH margin | ≥ 10–20% | Below margin → cavitation + impeller erosion. Maintain system pressure + de-air. |
| Footprint / clearance / weight | per model (e.g. 2400×1200×1200 mm · ~1793 kg wet) | Service clearance front/rear per manual; floor loading checked for filled weight. |
02 Water quality — acceptance criteria & standard limits
The single biggest driver of cold-plate reliability is fluid chemistry. Two loops have different limits — the secondary Technology Cooling System (TCS) that touches the chips, and the primary Facility Water System (FWS) — because the fluids differ (inhibited PG25 glycol vs treated facility water). Use the table that matches your loop as the commissioning acceptance criteria and the operational monitoring band.
STANDARD published code / spec figure · TYPICAL/VENDOR typical or vendor range, not a fixed standard
ASHRAE facility-water (W) classes — max supply temperature
ASHRAE TC 9.9 liquid-cooling classes. The number is the max facility supply temperature in °C; all classes share a 2 °C lower limit. Higher class → more free-cooling / chiller-less operation. STANDARD ASHRAE TC 9.9 (2021)
| Class (new) | Old | Max supply temp | Typical plant |
|---|---|---|---|
| W17 | W1 | ≤ 17 °C (62.6 °F) | Chiller + tower, water-side economizer |
| W27 | W2 | ≤ 27 °C (80.6 °F) | Chiller + tower, more economizer hours |
| W32 | W3 | ≤ 32 °C (89.6 °F) | Chiller-less in most climates |
| W40 | — | ≤ 40 °C (104 °F) | Chiller-less (added 2021) |
| W45 | W4 | ≤ 45 °C (113 °F) | Chiller-less for capex/efficiency |
| W+ | W5 | > 45 °C | Warmest-water designs |
TCS (secondary / cold-plate loop) — fluid acceptance criteria
For single-phase water-based cold-plate racks. STANDARD OCP, Guidelines for Using Water-Based Transfer Fluids in Single-Phase Cold-Plate Racks, Rev 1.0, Table 1
| Parameter | Acceptance limit | Notes |
|---|---|---|
| pH | 8.0 – 10.5 | Inhibited/glycol TCS runs alkaline; >9.5 may reduce biocide need. (Don't apply the older generic "7–8.5" water figure here.) |
| Conductivity | < 1,500 µS/cm @25 °C | After treatment, then monitor rate-of-rise. Inhibited fluid is intentionally conductive — the <5–10 µS/cm target is for DI-water loops only. |
| TDS (total dissolved solids) | < 1,000 ppm | — |
| TSS (suspended solids) | < 5 ppm op · < 1 ppm fill | Tighter at initial fill. |
| Total hardness (CaCO₃) | < 30 ppm · < 2 ppm fill | Scale prevention. |
| Chloride | < 50 ppm (304SS) · < 500 ppm (316SS) | Pitting risk — limit depends on wetted stainless grade. |
| Turbidity | < 5 NTU | — |
| Bacteria | < 1 CFU/mL fill · < 100 CFU/mL op | Biofilm clogs microchannels. |
| Corrosion inhibitor (azoles/TTA) | > 100 ppm | No nitrites. Depletes over time — dose per sample. |
| Cu / Fe ions | < 0.2 ppm / < 0.1 ppm | Corrosion indicators — rising = active attack. |
| Side-stream filtration | down to < 5 µm | Polishing loop keeps particulate low. |
TCS vs FWS — side-by-side limits
STANDARD ASHRAE TC 9.9, reproduced in Dell "Liquid Coolants Guidance for TCS and FWS", Table 2
| Parameter | TCS (PG25 secondary) | FWS (facility primary) |
|---|---|---|
| pH | 8.0 – 10.5 | 7.0 – 9.0 |
| Total hardness (CaCO₃) | < 50 ppm | < 200 ppm |
| Chloride | < 25 ppm | < 50 ppm |
| Sulfate | < 25 ppm | < 100 ppm |
| Sulfide | — | < 10 ppm |
| Cu / Fe | < 2 ppm each | — |
Make-up / top-off water (OCP sub-spec). STANDARD chloride < 25 ppm · sulfate < 25 ppm · Ca < 25 ppm · Mg < 25 ppm · total hardness < 50 ppm CaCO₃ · < 1 CFU/mL. Blend/dilution water benchmark: ASTM D1193 Type II (resistivity > 1 MΩ·cm, ≈ < 1 µS/cm). Not over-stated: TOC has no published numeric cold-plate limit (treat as TYPICAL/VENDOR); the biostatic glycol % is a freeze/viscosity choice, not a fixed biocide standard.
03 Installation & commissioning — numeric procedure
The numbers behind the §04 checklist ticks. Where a value is a published code/spec it is tagged STANDARD; where it is common engineering practice it is tagged TYPICAL/VENDOR so you don't over-cite it in a design package.
Pressure test (hydrostatic)
- Hydrostatic test pressure = 1.5 × design pressure, temperature-corrected PT = 1.5 × PD × (ST/SD).STANDARD — ASME B31.3 / B31.1
- Hold ≥ 10 min at test pressure before/while examining joints, then for the full inspection duration.STANDARD — ASME B31.x minimum (a 4-hr hold is a TYPICAL/VENDOR DC practice)
- Acceptance = no leakage / no visible decay. ASME does not set a "% drop over N hours" — any such figure is project/vendor.pneumatic alt: B31.3 1.1× · B31.1 1.2×
Flush & cleanliness
- Turbulent flush — sustain Re > 4,000 (≈ 2–3 m/s) to scour debris before connecting cold plates.STANDARD flushing convention · velocity TYPICAL
- Cleanliness target by ISO 4406 (>4/>6/>14 µm code) or NAS 1638; sensitive loops aim ~16/14/11 or better.STANDARD frameworks · target TYPICAL
- Flush until clean — common gate: conductivity stabilises < ~5 µS/cm (DI loops); TCS in-loop turbidity < 5 NTU (OCP fluid spec).VENDOR gate · OCP turbidity STANDARD
Filtration sizing
| Loop | Filter rating | Source |
|---|---|---|
| FWS / primary | 150 – 500 µm | TYPICAL/VENDOR |
| TCS / secondary (main) | 25 – 50 µm | 50 µm ≈ 288 mesh VENDOR (Lenovo Neptune citing ASHRAE) |
| Cold-plate side-stream | < 5 – 10 µm | OCP (< 5 µm) |
| Polishing (fine side-stream) | down to 0.2 µm | VENDOR (Boyd/Eaton) |
Coolant & flow
- PG25 (25 vol% propylene glycol) is the de-facto DTC grade (freeze ≈ −10 °C); practical band 20–35%.PG25 STANDARD (OCP-listed) · band TYPICAL
- Design flow ≈ 1.5 LPM/kW at ~10 °C rise; published band 1.25–2.0 LPM/kW (7.5–12 °C ΔT). Physics: Q = ṁ·cp·ΔT, cp(PG25) ≈ 3.9–4.0 kJ/kg·K.STANDARD — OCP cold-plate design point
- 2-phase / refrigerant DLC fluids: R-1233zd(E) (GWP 1, A1, boil ≈ 18 °C) · R-515B (GWP ≈ 293, A1) · R-1336mzz(Z).vendor datasheet figures
Quick-disconnects (QD) — OCP UQD / UQDB
- Match the QD standard across the loop — no mixed couplers. UQD = hand-mate, drip-free, hot-pluggable; UQDB = blind-mate with ±1 mm radial tolerance for rack/manifold insertion.STANDARD — OCP UQD/UQDB
- Sizes UQD02/04/06/08 · max working 100 psi (6.9 bar) · min burst 300 psi (20.7 bar) · op 17–65 °C · ≥ 5,000 mating cycles · dripless spillage ~0.02–0.07 mL/connect.cross-checked vs Parker/CPC datasheets
- No standardised coupler torque (push-to-connect). Torque applies only to threaded fitting/manifold ports — per vendor.TYPICAL/VENDOR
Leak detection & response
- Two sensing types: rope/cable zone sensing (area coverage under floor / along runs) + point/spot sensors (CDU drains, QD manifolds, headers).industry practice
- Response sequence: detect → alarm to BMS/DCIM → isolate / auto-shutoff valves → contain. Guidance: OCP OAI Liquid Cooling Guidelines + ASHRAE TC 9.9 5th ed.STANDARD guidance · timing figures TYPICAL
04 Installation checklist
Phased from delivery to handover. Tick each item; the printable form (§06) captures readings + sign-off.
Site preparation
- Floor loading verified for the CDU's filled, operating weight + service access.confirm point load vs raised-floor / slab rating
- Service clearances front/rear/sides per the unit manual.e.g. ≥1 m service aisle; door swing + tube routing
- Facility water available (L2L) with isolation valves, strainer, flow/temp at design point — or confirm L2A room-air budget.CDW supply/return temp + flow per data sheet
- Dual power feeds available (A/B), correct breaker rating + earthing.N+1 PSU; verify voltage/phase
- BMS / leak-detection integration points identified (Modbus/BACnet/SNMP).
Mechanical install
- CDU set + levelled, anti-vibration mounts seated.
- Primary & secondary piping connected; pipe spec/insulation per design; supports fitted.no strain on CDU ports; correct DN size
- Manifolds + quick-disconnects installed; QD standard matched (no mixed couplers).verify drip-less QD type per vendor
- Leak-detection rope/sensors routed at base, manifolds, QDs.
Fill, flush & commission
- Pressure test primary + secondary to spec; hold + record (no decay).test pressure + hold time per manual
- Flush secondary loop to remove debris before connecting cold plates.flush until clean / target turbidity
- Fill with correct coolant (PG% / DI / dielectric) and de-air the loop.record fluid type + concentration
- Set flow + dP + supply temp to design; verify against rack heat load.
- Leak-detection + alarm test; confirm auto-isolation behaviour.
- Redundancy test — fail one pump / one PSU, confirm N+1 hold.
- BMS points verified (flow, temps, dP, pressure, alarms reading correctly).
- Handover pack — as-built, fill record, commissioning sheet, O&M manual filed.
05 Routine inspection checklist
Walk-down checks. Anything outside the §01 band → log + see the symptom table (§05).
- No leak alarm; visually dry at base, manifolds, QDs, HX, pump seals.0 leaks · check for glycol residue / drips
- Flow within band.~1.0–1.5 LPM/kW
- Differential pressure stable (no clog rise).~0.5–3.0 bar
- Supply / return temps + ΔT at target.supply ~17–45 °C · ΔT ~8–12 °C
- System pressure in band; expansion / make-up healthy.~2–6 bar
- Pump status — duty running, standby ready (N+1); no noise/vibration.
- Filter dP within clean range; no clog warning.
- No alarms on HMI; data quality live (not stale).
06 Preventive-maintenance (PM) checklist
PM tasks by cadence. DAILY WEEKLY MONTHLY QUARTERLY ANNUAL. Intervals are a baseline — follow the vendor PM schedule and adjust to fluid-sample results.
| Task | Cadence | Acceptance / action |
|---|---|---|
| Read & log flow / dP / temps / pressure; check alarms | DAILY | All within §01 band; trend for drift. |
| Visual leak walk-down (base, QDs, manifolds, seals) | DAILY | Dry; any glycol residue → investigate. |
| Verify N+1 (standby pump/PSU healthy) | WEEKLY | Standby ready; exercise per vendor. |
| Check make-up rate & expansion-tank level/pressure | WEEKLY | Make-up minimal; level/pressure in band — rising make-up = slow leak. |
| Verify dew-point reset active (supply ≥ room dew point +2–3 °C) | WEEKLY | No condensation risk; RH sensor reading true. |
| Leak-detection function test | MONTHLY | Alarm + isolation confirmed. |
| Filter dP check / replace if clogged | MONTHLY | Replace at vendor dP threshold; record. |
| Coolant sample — pH, conductivity, glycol%, biology, particulate | QUARTERLY | Within ASHRAE W-quality; dose/replace per result. |
| Flow & dP sensor calibration check | QUARTERLY | Within sensor tolerance; re-cal if drift. |
| Pump seal / bearing inspection (vibration, noise) | QUARTERLY | Service per runtime hours / vendor. |
| Control-valve stroke / actuator demand-vs-feedback test | QUARTERLY | Deviation <10%; fail-safe (bypass) confirmed. |
| BMS/DCIM telemetry & alarm-point verification | QUARTERLY | All points live + correct; no stale data; alarms route. |
| Corrosion-inhibitor reserve + Cu/Fe ion test | QUARTERLY | Inhibitor >100 ppm; Cu <0.2 / Fe <0.1 ppm (rising = active corrosion). |
| Heat-exchanger approach (ATD) check / clean | ANNUAL | Rising ATD = fouling → clean side. |
| Coolant replacement (or per sample) | ANNUAL | Replace if chemistry out of band. |
| Full redundancy + failover drill; firmware/controls review | ANNUAL | N+1 holds under fault; backups current. |
07 Symptom → cause → action
Troubleshooting reference. On any active symptom: protect the IT load first (the CDU's N+1 / isolation logic), then diagnose.
| Symptom | Likely cause | Immediate action | Corrective action |
|---|---|---|---|
| High differential pressure | Filter clogging; valve throttled; air pocket | Check filter dP; confirm valves open; bleed air | Replace filter; re-balance; full de-air |
| Low / no flow | Pump fault; closed valve; QD not seated; airlock | Confirm standby pump took over (N+1); check valves/QD | Service/replace pump; reseat QD; purge air |
| Leak alarm | Loose QD; failed seal/gasket; manifold/HX leak | Acknowledge → isolate affected zone; locate via rope/zone | Tighten/replace QD or seal; pressure-test; refill + de-air |
| High coolant supply temp | Facility water warm/low flow (L2L); HX fouling; high ATD | Check primary supply temp/flow; check ATD | Restore CDW; clean HX; verify control setpoint |
| High room temp (L2A) | Room CRAH can't absorb rejected heat; coil dirty | Check CRAH capacity/airflow; clean coil | Add room cooling / move to L2L; coil service |
| Rising ATD (approach) | Heat-exchanger fouling; reduced flow one side | Compare primary/secondary ΔT; check flows | Clean/flush HX; restore flow |
| Pump fault / vibration | Bearing/seal wear; cavitation (low pressure/air) | Switch to standby; raise system pressure; de-air | Replace pump/seal; correct expansion control |
| Coolant chemistry out of band | Biology growth; glycol degradation; contamination | Sample; dose biocide / inhibitor | Partial/full coolant replacement; side-stream filtration |
| Air in loop (gurgling, unstable dP) | Incomplete de-air; make-up drawing air; leak | Bleed at high points; check make-up | Find/seal air ingress; re-fill correctly |
| Stale / lost telemetry | Sensor fault; comms loss to BMS | Treat values as suspect; local gauge check | Replace sensor; restore comms; re-cal |
| Condensation / sweating on plates & pipe | Supply temp driven below room dew point; dew-point reset disabled / RH sensor fault | Raise supply setpoint above room dew point; check RH sensor | Enable dew-point-aware reset (≥2–3 °C above dew point); fix humidity sensor; insulate |
| GPU throttling / hot sleds | Flow maldistribution across the manifold; under-provisioned flow vs chip TDP | Compare per-branch flow; verify total flow ≥ design L/min/kW | Balance manifold (balancing valves / flow restrictors); raise flow setpoint |
| Two-phase cold-plate dry-out (2-phase) | Vapour quality too high in hot sleds; uneven 2-phase distribution | Check refrigerant charge / sub-cooling; inspect distribution | Re-balance flow restrictors; restore charge; correct manifold geometry |
| Frequent make-up / falling level | Slow leak; air being purged; expansion bladder fault | Trend make-up rate + level (LT); leak walk-down | Find/seal leak; recharge / replace expansion bladder; re-pressurise |
| Control valve fault (TCV) | Actuator demand vs feedback >10%; signal loss → fail to bypass = no cooling | Switch to manual / verify fail-safe position; protect IT load | Re-stroke / replace actuator; re-tune PID (Ziegler-Nichols) |
| System over-pressure | Thermal expansion with closed expansion path; PSV not relieving | Verify expansion/relief path open; check PT vs component PN | Restore expansion control; test/replace PSV; re-set fill pressure |
08 CDU service record — printable form
Fill on-screen and Print / Save as PDF. Captures site identity, readings against the §01 band, checklist pass/fail, and sign-off. Stored on the RZ site as the standard CDU service-record form.
Readings vs target band
| Parameter | Target band | Reading | Pass / Fail |
|---|---|---|---|
| Supply temp (°C) | ~17–45 | ||
| ΔT (°C) | ~8–12 | ||
| Flow (LPM/kW) | ~1.0–1.5 | ||
| Differential pressure (bar) | ~0.5–3.0 | ||
| System pressure (bar) | ~2–6 | ||
| Coolant pH / cond. / glycol% | ASHRAE W1–W4 | ||
| Filter dP / status | clean range | ||
| Leak detection | 0 leaks | ||
| Pump N+1 (duty/standby) | both healthy |
Disclaimer. Independent educational reference built from publicly available vendor pages and ASHRAE TC 9.9 / OCP guidance. Parameter bands are generic baselines — confirm exact limits, fluid spec, torque/pressure values and PM intervals against the specific CDU's manufacturer manual and your IT hardware spec. Not procurement, safety, or engineering advice; validate with the vendor and a qualified mechanical engineer.