Dive Brief:

• Colonial Pipeline was back online less than a week after a May 7 ransomware attack forced the refined products system to shut down, and the company credits its response procedures, rapid outreach to law enforcement — and payment of the roughly $4.4 million in bitcoin ransom — as keys to the swift operational recovery.
• While electric utilities and pipelines face different challenges, experts say power companies can learn from Colonial's response. But the question of whether or not to pay a ransom remains an open debate.
• "Ransomware puts organizations in an impossible situation," Sandra Joyce, executive vice president at Mandiant Threat Intelligence, said in an email. The company is one of three consultants Colonial called on to help address the attack.

Dive Insight:

Colonial officials say their first call, in the hours after discovering the attack, was to the Federal Bureau of Investigation. That in turn set off an information-sharing mechanism that ultimately brought more than a half dozen other agencies and entities to the table.

Colonial Pipeline President and CEO Joseph Blount Jr. testified Tuesday before the U.S. Senate Committee on Homeland Security & Governmental Affairs, revealing that the company also "worked closely with the White House and National Security Council, [and] the Department of Energy, which was designated as the lead federal agency" on the incident.

Blount said other agencies involved in the response included: the Department of Homeland Security, the Pipeline and Hazardous Materials Safety Administration, the Federal Energy Regulatory Commission, the Energy Information Administration, and the Environmental Protection Agency.

The FBI also looped in the Cybersecurity and Infrastructure Security Agency, said Blount, though the company has taken criticism for not reaching out to CISA directly. 

"Private industry can't do it alone," Blount told lawmakers. "The partnership with government is very important."

Colonial's first contact was to the FBI's Atlanta office, though eventually the San Francisco office took over, said Blount. He conceded that the company did not follow the federal law enforcement agency's official position on ransomware — which is not to pay — though he added that conversations about whether to pay did not come up in the first day's discussion with law enforcement. 

The decision to negotiate with the hackers was made the evening of May 7, said Blount, and payment was made May 8. Colonial hired legal and negotiation experts, said Blount, to deal with the hackers and to ensure the company was not making payments to an entity under sanctions by the U.S. Treasury.

"I made the decision to pay, and I made the decision to keep the information about the payment as confidential as possible," Blount said. "It was the hardest decision I've made in my 39 years in the energy industry."

Blount defended the effort, saying it "was the right choice to make" given that a prolonged pipeline outage could mean shortages of gasoline and jet fuel.

Ransomware is a growing threat, and there have been other instances of utilities paying. The Lansing Board of Water & Light in Michigan paid a $25,000 ransom five years ago to unlock some of its systems.  

The U.S. government says it has recovered about half of the ransom Colonial paid, which Blount also credited to the company and government's rapid response. Federal authorities have been "true allies," he said.

Colonial also brought in three outside consultants to help it respond to the attack — Dragos,  Mandiant Threat Intelligence and Black Hills Information Security.

Mandiant's Joyce says the issue of ransomware is a challenging situation for victims, and the company has no position on whether to pay or not.

The Edison Electric Institute, which represents U.S. investor-owned electric utilities, says it also does not have a position on whether its members should pay a ransom. However, the group's members have worked with the Electricity Subsector Coordinating Council to develop guidance, including issues to consider before making a payment.

"Paying the ransom may encourage this criminal business model," the guidance warns.

Security experts warn that paying a ransom is risky: there is no guarantee a company will get the keys to restart its systems, or that those decryption keys will work, or that hackers will not simply strike again. In Colonial's case, however, Blount said the keys provided by hackers were in fact useful, though not perfect. The company is still in the process of bringing online seven financial systems that have been unavailable for more than a month.

"What a lot of people don't realize about cyberattacks and the repercussions, is it takes months and months and months ... and in some instances we've heard years, to restore your systems," said Blount.

Electric utilities should examine both Colonial's response to the attack and the fallout, said Rick Tracy, chief security office at cloud security provider Telos Corp. "It’s not just about how you respond, it’s about how you prepare," he said in an email.

"In my opinion the power grid is a nightmare scenario," Tracy said, recommending utilities "learn from these past attacks. Implement essential controls. Also, have incident response plans ready to go in the event there is a ransomware attack."