People in the events industry are used to planning for the worst. “But you never really expect it to happen,” said Kurt Hoff, information technology director at the Thomas & Mack Center.
But that "worst," at least for the person in charge of ensuring producers of events at the T&M have the network access they need, did happen just as the venue was preparing for its biggest event ever. Two nights before the country would tune in to watch the final matchup between the candidates, a router — a critical component for any network — had a catastrophic failure.
The T&M’s Mark Horn put the massiveness of the operation in perspective. He had to install some 850 telephone lines throughout the buildings and into the parking lot. “I’ve been here 21 years and this is the biggest thing, by far, we’ve ever done,” he said. “Normally, for something like the NBA All-Star game or boxing, we work with a single production company — an HBO and maybe one pay-per-view truck.”
This time, the teams had to provide power, communications, and data lines for 20 trailers for major broadcasters plus hundreds of smaller news organizations. All told, they ran 11 miles of data and voice lines. “Those trailers are really complexes unto themselves,” Hoff said. “And we’ve never had to run lines outside building like that.”
Fortunately, Hoff was able to turn to a newfound, cross-organizational team to fix the problem. As a self-supporting unit, the T&M typically operates independent from 51ԹϺ’s office of information technology (OIT). But the debate was so much bigger than either organization could handle on its own.
Robert Cray, senior network engineer in OIT, had designed the debate’s network to have redundant and duplicate connectivity, so service wasn’t disrupted when it switched to the second router. But what if that router failed just as the debate began?
The team of combined teams jumped to the challenge. They managed to get the replacement flown in and Cray was able to configure the router just before the Secret Service lock down, which required everyone to leave the building for security sweeps.
“Yes, we could have flown the plane with one engine, but I will sleep tonight knowing we have two,” Hoff said just 24 hours before the debate began. “We literally had dozens of people engaged to resolve this one issue … all stepped up to the plate and made what could have been a international black eye for 51ԹϺ turn out to be something that only us, on the back side of the cameras, know about.”
The two groups began working together a year before the debate to design and setup a complete, though temporary, network infrastructure. “We may have seen each other in meetings every now and then before,” Hoff said, “but this was completely different. This put us in the trenches together, really battling the challenges. We became battle buddies.
Chief among them, he said, was Lori Temple, vice provost for information technology. “She engaged with us early. As she really saw the immensity of it unfold, she’d loop in a Robert and then a Lorita and a Cam from her team,” he said referring to Robert Cray, Lorita Chesler, and Cam Johnson.
The debate offered Temples’ team the change to enhance their skills. Unlike OIT’s regular job supporting and improving campus technology, the debate’s infrastructure was temporary.
“This is not our venue and being part of an event like this is outside our comfort zone,” she said. “Our project management types have had to work very hard in an environment in which we’re out of control.
“But we’re good at procedures so they’ve adapted really well.”
Like when they changed their carefully planned order-taking process midstream. Naturally, the staff had been inclined to turn to technology to manage the process. “In the end, they said, ‘This is just too complicated. We’re going to use paper and bins.’”
