The expense had been massive. The coordination effort, given all the constraints, a miracle of planning. The accuracy and effectiveness of the planning, all done on paper, was a testament to the intelligence of the men and women involved. The scene of Sean’s house, their temporary base of actions, covered in paper and flip charts and hand-drawn timelines, recalled great accomplishments of the mid-twentieth century, when humans routinely tackled tremendous efforts in nothing but shirtsleeves and paper charts.
Human intelligence, creativity, and planning had prevailed. They won!
Chapter 16
“Helena, have you seen this?”
Helena looked up at her shift partner, Jan. They sat in the monitoring room of Europe’s most secure data center. Located in a converted underground bunker in Stockholm, the massive computer facility was fit for a scene from a James Bond movie. At just over 4,000 square feet, the concrete and stone tomb contained tens of thousands of servers and hard drives. Designed to be secure against a nuclear bomb, and using retired submarine engines for backup power, it even contained an independent air supply, kitchen, food stocks and office space for the administrators on duty. Armored steel doors protected against mere human incursions.
It was staffed twenty-four hours a day, three hundred and sixty-five days a year by specially vetted system administrators so that any issue could be addressed ASAP for the clients who paid for the privilege of hosting their data and web applications in the elite data center. The sysadmins worked in a glass-enclosed room with a separate air filtration system that overlooked the entire datacenter.
For Jan and Helena, it was just another day at work.
“Have I seen what?”
“My sandwich. Look, those idiots at the grocery put mustard on my sandwich. I never eat mustard on my sandwich.”
Helena sighed, and took a sip of her coffee. She went back to reading the book she’d brought that day, the latest sci-fi novel by some writer from Scotland.
“Holy shit, now look at this,” Jan cried out.
“No.”
“No, really look.”
“I don’t care about your sandwich,” Helena said, forcing her eyes to remain on her book.
The shrill beeping alarm seconds later drew Helena’s attention, and she looked up to where Jan was staring, dumbfounded, at the monitor board.
Jan pointed at the indicators on the sixty inch display hanging above their heads on the wall. “We’ve been humming along at thirty percent of processor capacity all morning, and now we’re running above ninety percent across the board. That leaves us with almost no spare processor power in case anything else peaks. And our bandwidth was running at about twenty percent of capacity all morning, and now it’s gone up to almost eighty percent of maximum capacity. What is it? A denial of service attack?”
As Jan spoke, Helena could feel a shift in the vibrations of the facility, as cooling fans were automatically sped up by the monitoring system in response to the higher processing load.
Helena paused to consider his suggestion. A denial of service, or DOS, attack was a technique used by hackers to bring down Internet service providers who hosted web servers on behalf of clients, or corporations running their own web servers. The hackers used thousands or millions of PCs that had been compromised by specially designed computer viruses. Those compromised systems formed a virtual army of slave computers that could be used to send email spam, or launch a DOS attack.
“Let’s look at the traffic before we jump to conclusions.” Helena set her book down. She silenced the alarm and started working with her primary computer to see which programs running on the servers accounted for the jump in CPU consumption, while she simultaneously worked on a second computer to look at the network traffic to see what accounted for the jump in bandwidth use.
“What the hell?” Helena looked puzzled. “This load is all being generated internally. Look, at 2500 hours, we launched an application simultaneously on all servers, on behalf of account 6502530000. That account is…” Helena paused while she looked up the record in the customer database. “That’s on behalf of Avogadro. Let’s see what their account says…”
Jan eagerly looked over her shoulder. He had started only a few weeks earlier, and he found it thrilling to watch a master admin like Helena navigate her way through myriad control and monitoring systems they used to administer the computers. Surrounded by two large displays, and her personal MacBook Pro on the side, she had dozens of applications open, monitoring everything from accounting databases to system logs to router dashboards. Before Jan could grok what Helena was doing with one application, she would move on to the next. His head started to hurt.
“We have a service level agreement in place to give Avogadro top preemptive priority. It looks like they must have wanted an emergency backup in case their own data centers were affected.”
Even Jan knew that Avogadro had more computer servers than any other company in the world. “Why would they want to use us? Don’t they have hundreds of their own data centers around the world?”
“Yes, but maybe they were anticipating a problem,” Helena responded. “According to this, we signed the contract with Avogadro just a few weeks ago.”
“So what are we running? Their email servers? Their search engine?” Jan wondered aloud.
“Well, it doesn’t look like we’re running any customer facing applications. If you look at the traffic profile,” and here Helena gestured to the second display. “You can see that the majority of the traffic is outbound. Looking at the ports and addresses, it seems like the Avogadro code is sending a ton of emails, and big ones too. They are getting some emails inbound, but not enough to account for all of their customers. It’s puzzling for sure. Could they be remotely restoring their servers via email?” Helena shook her head at the improbable notion.
She turned to the third computer on her desk, her personal Mac. “Let me see what happens when I visit Avogadro.” She launched two web browser windows, going to the Avogadro search page in one, and her personal Avogadro email account in the other. “Both email and search web servers are returning a not reachable error. That must mean Avogadro has a major outage.”
“What do we do?” Jan asked.
Helena paused and thought for a moment. “The application and traffic is legitimate. Avogadro paid us for top priority, including the ability to preempt anything else we’re running. They wanted this application, whatever it is, to run in the event that they had a major outage at their own data centers. I can’t peek at the actual code or traffic without violating our customer privacy policy. So I think we just babysit it and hope the servers don’t melt down under the load.”
She glanced at the load indicators, which showed that processing load was pegged at a hundred percent. Looking out the glass window of their enclosure, she glanced across the datacenter floor to see that every blinking indicator light on every rack-mount server and every router was a solid red. She’d never seen traffic loads like this.
“Look, I think there’s a few spare racks that aren’t powered up yet,” Helena said, heading for the door. “I’m going to turn on every spare piece of hardware I can find. I want you to go into the admin tool and throttle back any application that isn’t Avogadro. We’ve got to free up some capacity here.”
Helena headed out the door into the main room.
Jan swallowed hard, and sat down in front of Helena’s computer. His hands trembled slightly as he rested them on Helena’s keyboard. He summoned up his courage and got to work.
It was the third day since the attack that took ELOPe down. Across Avogadro, everyone was working around the clock to restore services and data. With no opportunity to alert the company ahead of time to the outage and with communications largely down, the best the Emergency Team could do was to have a point person at each site who had instructions on the proper process to restore computers to known good backups, backups free of ELOPe,
