Renaming a directory broke the inference server
The new inference build compiled correctly. It supported both generations of accelerator in the farm. It lived beside the old build so the working version remained available.
Then I gave the new directory the old directory’s name, and every model launch failed.
No model changed. No GPU changed. No configuration file changed. The deployment failed because I moved a folder.
That sounds like an operating-system oddity. It was actually an engineering assumption that had never been written down.
The change looked routine
The farm combines Volta-generation V100s with an Ampere-generation RTX 3090. The
llama.cpp build therefore had to target both accelerator architectures:
cmake -B build-v2 -DCMAKE_CUDA_ARCHITECTURES="70;86"Building into build-v2 was deliberate. The existing build still served models, and the
new one could be compiled and inspected without disturbing it. That is the safe half of a
parallel replacement: keep the known working state intact while the challenger is being
prepared.
The mistake came during the apparent cleanup. The old build directory was moved aside,
and build-v2 was renamed to build so the surrounding launch configuration could keep
using the familiar path.
The binaries were identical before and after the rename. Their location was not.
Every model spawn failed. Reversing the rename restored the path the executables expected, and the launches worked again. That isolated the change more effectively than another round of speculation would have.
The executable remembered where it was born
A compiled program often depends on shared libraries loaded when it starts. A build-tree executable can carry an RPATH—simply a list of library locations stored inside the executable itself.
In this build, those locations were absolute. The binary did not remember “find the libraries beside me.” It remembered the full path to the directory where it had been built.
Renaming the directory changed the filesystem but did not rewrite the path already stored inside the executable. The launcher found the binary at its new name; the binary then looked for its libraries at the old name. The files existed, but not where the binary had been told to look.
The build directory was not packaging around the artifact. Its path was part of the artifact.
The useful evidence was the rollback
The first temptation in a failure like this is to widen the investigation: CUDA versions, driver compatibility, model configuration, permissions, or the launcher itself. All were plausible. None explained why the same compiled files worked before the move and failed after it.
The reversible deployment supplied the decisive comparison:
- The new build worked in the directory where it was created.
- The same build failed after that directory was renamed.
- Restoring the original path restored operation.
That is stronger than a fluent diagnosis. It is a controlled difference with a repeatable result.
The old build remaining available mattered for another reason: diagnosis did not have to compete with an outage. A failed candidate could lose immediately while the previous working state stayed intact.
The durable rule
The deployment procedure now treats a compiled build tree as immovable:
- Configure a new, uniquely named build directory.
- Compile and verify the binaries in that directory.
- Leave the verified directory where it was built.
- Repoint a stable symbolic link to the new directory.
- Keep the previous link target available for rollback.
The stable name moves. The artifact does not.
There are other legitimate solutions. A formal installation step can produce relocatable artifacts, and relative library paths can be designed deliberately. But a build tree should not be assumed portable merely because every file moved together.
The verification also has to occur through the final operating path. “The build completed” proves compilation. It does not prove that the service can launch the binary, resolve its libraries, load a model, and use the intended accelerators after deployment.
Why this belongs with the hardware failures
The farm’s physical failures involved connectors that fit while carrying the wrong voltage, ports reported as available while their lanes were not routed, and a damaged CPU socket that first presented as a cooling problem. This software failure has the same shape.
The visible object suggested one thing. The system’s actual dependency said another.
A directory looked like a container. It was part of the executable’s state. A rename looked reversible. It was safe only because the prior state had been preserved and the comparison was immediate.
Experienced engineering does not eliminate wrong assumptions. It makes them inexpensive to disprove, keeps the working system close enough to recover, and turns the discovered assumption into a rule that outlives the incident.
That is the real result of this failure. The inference server came back quickly. The next build no longer has to rediscover why.
The hardware, routing architecture, and the other failures behind this lesson are documented on The AI Farm. The wider acceptance method is in Judgment Is the Control Plane.