Buying cheap datacenter hardware means inheriting every assumption the people who
designed it made about the datacenter it would live in. None of the following is in
a build guide. Each one cost real days, and each one turns out to be the same lesson
the software half of my work keeps teaching.
The two-week fault
A single bent pin in the CPU socket
The server board never completed POST. It parked at the same diagnostic code
through two different CPUs, two thermal-paste jobs, a firmware upgrade and a BIOS
upgrade. The management controller had been logging thermal-trip events, so the
first two weeks went into cooling — and the cooling was fine. After the firmware
update the failing boots logged nothing at all, and the controller showed the CPU
temperature sensor and all eight memory temperature sensors as disabled. That was
the real signal: the management sideband between the controller and the CPU runs
through the socket, and it had gone silent. Thermal was the symptom. The disease was
one flattened pin in the first row under the frame wall, drawn out into a gold
ribbon lying across its neighbour.
An error that reports a plausible cause will absorb every hour you give it.
The sensors that went quiet were worth more than the alarm that kept firing —
absence is evidence, and almost nothing is built to alert on it.
The photograph that lied
Raked light, or you will not see it
Photographed from near-vertical, the damage vanished. A flattened pin presents
no tip to the camera, so it renders as a slightly dark segment of an otherwise
bright row — which reads as "nothing there." It only glints under raked light, and
it took two independent oblique angles to establish that the anomaly was real and
not an artefact. A canned-air puff from a distance ruled out the other candidate:
a stray fibre would have blown away. It did not, so it was metal.
Design the test so that a negative result means something. "I looked and
didn't see it" is not a measurement if the method cannot resolve the defect.
The card killer
Two 8-pin connectors that mate and must not
These accelerators take an EPS (CPU-style) 8-pin. Graphics cards normally take a
PCIe 8-pin. Same appearance, physically matable, different wiring — four 12 V pins
versus three, in different positions. Feed one the wrong way and 12 V lands on the
card's ground pins: instant death, not covered by warranty. The keying is moulded
into the card and cannot be wrong, but a no-brand adapter can have perfect keying
with wires crimped into the wrong cavities. It mates, it photographs correctly, and
it kills the card.
My first instinct was to check that the adapter ran each pin straight through
to the same position. That is exactly backwards — a straight-through cable IS a PCIe
cable. The adapter's whole job is to move the conductors. The thing to verify is
never "do the ends match," it is "does the voltage land where the load expects it."
The meter
Silence has to mean something
The test is continuity from a known-good ground to each of the eight pins, using
the power supply's bare chassis and the card's own bracket as references — count the
beeps. Four means EPS and correct; five means PCIe wiring and stop. The step people
skip is the self-test: touch the probes together first and confirm the meter beeps.
A dead battery gives silence on every pin, and silence reads exactly like "no
connection."
Any instrument whose failure mode looks like a passing result has to be
proven before it is trusted. This is the multimeter version of a test suite that
passes because it never ran.
The power supply
Pigtails, and cables that mate across brands
Modular PSU cables are pigtailed — one lead ends in two connectors. Using both
heads of a single cable to feed one 250 W card pushes that current through copper
rated for 150 W. Correct is one head from each of two separate cables. Worse: two
manufacturers' modular cables physically mate at the PSU end while carrying entirely
different pinouts, with no key to stop you. Nothing about the fit warns you. And the
cables that shipped in the box with the cards were server-riser cables, not
ATX-safe, despite being labelled for a GPU.
A connector that fits is not a connector that is correct. Every place this
build could destroy hardware, it did so through a joint that mated perfectly.
The ports that were not there
"Available" meant "never wired"
An external-GPU bring-up failed with no card enumerated. The dock was fine, the
cable was fine. The system inventory listed both external ports as "Available" —
which reads as "present and empty," but actually meant the lanes had never been
routed to them. They are disabled in the board's factory jumper configuration, and
the fix is two jumpers documented on page 26 of the manual.
Read the status word the vendor actually defined, not the one it resembles in
English. An hour with the manual would have beaten a day at the bench.
The physical constraint
The card does not fit — and that made it movable
On the workstation, none of the three slots could take a second full-size card:
the only CPU-attached x16 was occupied, the middle slot turned out to be a
closed-ended x1 that a x16 edge physically cannot enter, and the bottom slot was
long enough but collided with the power-supply shroud. So the fifth V100 lives
outside both cases, on an OCuLink dock over a x4 link. The cable can place that
32 GB card beside the four-V100 server when a job needs a 160 GB pool, or beside the
24 GB RTX 3090 workstation when a smaller host needs 56 GB. The link is a quarter of
the width — and it barely matters, because splitting a model across cards passes only
small activations between them. The one real cost is loading weights off disk, once
per load.
Measure what the bottleneck actually is. A specification that looks four
times worse can be irrelevant to your workload, and finding that out is what lets
you use hardware other people have ruled out.