Every retelling of the Large Hadron Collider's bad autumn gets one line: nine days after its first beam, an electrical fault wrecked the machine; fourteen months passed; it came back. The line is true. It is also a compression. Here it is at its actual length.
The first twenty seconds
19 September 2008, 11:18:36 CEST. The dipole circuit of sector 3-4 was ramping at 10 A/s toward its nominal 9.3 kA when, at 8.7 kA, a resistive zone appeared in the electrical bus between dipole C24 and quadrupole Q24. The first sign was 300 mV of voltage above the noise floor, in a circuit where none belongs. At 0.39 s it had grown to 1 V; at 0.46 s the power converter tripped and the energy discharge switch fired, both exactly as designed.
Within the first second, an electrical arc punctured the helium enclosure. At 3 and 4 seconds, the beam vacuum degraded in the two beam pipes. The relief discs on the vacuum enclosure opened on cue, and were still unable to hold the pressure below the design value of 0.15 MPa in the central subsector. The resulting forces knocked quadrupole cryostats off their support jacks at Q23, Q27 and Q31, broke their anchors in the concrete floor, and pulled dipoles off their internal supports. Six of the sector's fifteen tonnes of helium left the cold mass, two tonnes fast and four more slowly. Soot from the arc reached the beam pipes.
Nobody was hurt. CERN's interim report, filed twenty-six days later, uses the sentence I keep coming back to: “the safety systems performed as expected.” The failure was one connection. The damage was the consequences of it.
What got written down, and what CERN refused to claim
The 16 October analysis confirmed a faulty electrical connection between two magnets. The follow-up report of 5 December adds the honest part: the primary evidence was destroyed in the incident, so “the precise origin of the fault will always remain speculative.” Investigators reproduced the class of defect in the laboratory by applying large non-conformities to sample splices. A scan of the other sectors found no repeat among connections made in the tunnel, but two joints built in industry measured far outside the expected range, having passed their individual tests anyway. And one more line worth keeping: the same fault in a normal-conducting machine is “a matter of days” to fix. In a superconducting one it costs you the sector.
What the months were spent on
Warm sector 3-4 to room temperature. Open 66 interconnections. Endoscope the whole continuous cryostat for soot and insulation chips. Remove 53 magnets, 39 dipoles and 14 quadrupole sections, then clean or rebuild them and cold-test each one at nominal current before it goes back.
Then the hardening, which the one line never mentions. Some 2,000 new electronics crates and 160 km of signal cable, for a quench protection system fast enough to catch a splice going bad. Relief capacity increased eightfold at the quadrupole cryostats and, via new ports cut into every dipole cryostat, rising toward 40-fold overall, sized to swallow a helium discharge twice the size of the one that beat the old design. Anchors strengthened. And a new scheme to read splice resistance non-invasively, at 80 K, so nobody has to warm up a whole sector to learn that one splice is ill. The ring holds some 10,000 splices; the repair's real product was being able to read them.
The schedule, which never stopped moving
20 September 2008: “a minimum of two months down time.” October: restart in 2009. December: cold and ready for powering by the end of June 2009, physics by summer. 19 June 2009: on track, “about 2-3 weeks later than originally foreseen.” 16 July 2009: vacuum leaks at feedbox joints in sectors 8-1 and 2-3; both ends partially warmed to fix them; beam injection now “mid-November.”
It ended like this: operating temperature of 1.9 K on 8 October 2009; injection on 23 October; beam steered through three octants on 7 November; beams circulating again on 20 November, at ten in the evening. First collisions on 23 November, ATLAS recording its first candidate at 14:22. The 7 TeV collisions came 30 March 2010.
That is the between, itemized. Not a repair of one thing: a rebuild of the machine's reflexes, by hand, with teams borrowed from labs around the world, on a schedule that only looks decided in hindsight.
Receipts
- Incident in LHC sector 3-4, 20 September 2008: first estimate and the “matter of days” comparison.
- Interim summary report, 15 October 2008 (CERN EDMS 973073): times, sequence, helium quantities.
- CERN releases analysis of LHC incident, 16 October 2008.
- LHC to restart in 2009 and its follow-up report, 5 December 2008: 53 magnets, classes, consolidation numbers.
- CERN reports on progress towards LHC restart, 19 June 2009: 10,000 splices, 80 K measurement, QPS.
- News on the LHC, 16 July 2009: the two leaks, mid-November.
- The LHC is back, 20 November 2009, and first collisions, 23 November 2009.
Have a claim you don't trust? First look free: [email protected]