Breaking Down Building System Threats and Cybersecurity for CRE: 92 Days to Recover

March 14, 2022 | By: Fred Gordy, Intelligent Buildings, LLC
FacebookTwitterLinkedIn

Building cybersecurity is a major issue affecting thousands of commercial properties. Understanding the importance of cybersecurity is integral to preparing for cyber-attacks, and these examples give an idea of the consequences a cyber-attack can have. In this article, Fred Gordy, Director of Cybersecurity with Intelligent Buildings LLC, gives real-world examples of incidents that his company has experienced. These stories are based on actual events with modifications to mask the identity of those affected.

Event Type: Hack

A day before this event, a building engineer checked his personal email on the application server. He received an email that appeared to be from a fellow engineer. The building engineer that checked his email thought it was unusual that the employee had sent an email to him to his personal email, but the email had a link to a site that appeared to be something that might have come from this employee. The link did not take the engineer anywhere. It just appeared that the other employee had not copied the link correctly. The building engineer talked to the person that sent it later in the day and told them the link he sent didn’t work, for which the person let him know they had not sent him anything. Additionally, he advised that if he had sent it, the email would have gone to his work email because he didn’t know his personal email. None of these inconsistencies caused the engineer to notify anyone about what happened. They did, however, back up the application server to an external hard drive just in case. The day ended without incident.

About mid-morning the following day, another engineer needed to make setpoint adjustments because tenants on the 10th floor complained that their area was too cold. The engineer went to the application server to make the adjustments and noticed that a window was opened he had never seen. There was a message that said: “Your Important files are encrypted. Many of your documents, photos, videos, databases, and other files are no longer accessible because they have been encrypted. Maybe you are busy looking for a way to recover your files but do not waste your time. Nobody can recover your files without our decryption service.” There were also instructions on how to recover and how to pay for the recovery. There was also a countdown timer letting them know that all their files would be lost.

The engineer that clicked the link the day before was also in the room at the time and let the other engineer know that he had made a backup the day before and that they should call the vendor and have them reinstall the operating system and the application. The vendor was able to reinstall the operating system and installed the application and the files necessary to get the application up and running. This took a couple of days.

Not too long after the system was back up and running, there were some anomalies that occurred with some of the equipment in the central plant. The variable frequency drives (VFDs) seemed to be sometimes running slightly faster and sometimes slightly slower; however, this settled out and did not occur for the rest of the day.

Several days later, staff arrived and went about their typical day. The first indication of a problem was the engineer noticed that the central plant controllers were offline. They also noticed no alarms were showing for the central plant which there should have been at least the alarms associated with the offline state of the controllers.

Investigation of the plant found the chillers were off. Further investigation found that several of the VFDs were inoperable. The staff attempted to restart the main chiller manually but were not successful. They next tried to restart the other chiller manually and were unsuccessful. They began investigating mechanically the cause, at which time they found extensive damage to the pumps due to what they believed was cavitation.

At this point, the association between the ransomware attack was not even considered.

Identifying the Issue

It was determined that several of the pumps were going to need to be replaced, and each of the chillers required a full inspection to find out if there was any damage to them. The controls vendor was also contacted to investigate the system to check the application out and get the controllers back online and determine why alarms did not go out.

The staff determined that several of the pumps and VFDs needed to be replaced. They found that the VFDs appeared to have been run beyond their critical speed. The vendor happened to be listening and informed the engineers that the only way this could occur is if someone disabled the safety feature that would prevent it.

This changed the thought process from possible equipment failure, not likely due to the number of devices affected and the extent, to the system’s possible hack. The direction of the investigation took the focus that this was probably what happened, but how?

The first thought was that another unrelated hack had happened, but this seemed unlikely because lighting doesn’t strike twice in the same spot, right? The vendor went through the backup made the day before the attack and found that infection occurred that day of the backup, and it most likely came from the email that the engineer clicked when he checked his email on the application server.

Now, most of what is stated here are strictly speculation because there was no forensic data. Because the system had been reloaded and no logs were retained, who and when they accessed the system and what was done is still unanswered to this date.

Here is what is believed to have occurred:

  • The vendor found that a remote access trojan (RAT) was also in the payload of the ransomware.
  • Once the RAT was embedded and the hacker or hackers planted the ransomware, there was most likely a beacon that notified the hackers that the RAT was installed.
  • When the system was reloaded, the beacon notified the hackers that it was active again, and they realized they would not get paid, so they decided to damage parts of the system.
  • They remotely accessed the system and played with the VFDs to see what they could do but waited until they felt no one was watching the system and entered it sometime after 7 p.m.
  • They disabled the alarms.
  • They attacked the VFDs, which most likely caused cavitation, and destroyed the pumps.
  • They also disabled the central plant controllers.

The Results

The damage to the system included:

  • Roughly half the VFDs needed to be replaced.
  • The chillers had to be dismantled to inspect for damage.
  • Several pumps had to be replaced due to the damage from cavitation.
  • The central plant controllers had to be replaced because the hacker rendered them unrecoverable.
  • A new PC for the application was purchased to ensure no residual infection was present.
  • Because the backup was corrupted and the only backup was a year old, the application host programming had to be partially redone.
  • The entire process to fully recover took 92 days and thousands of manhours.
  • During that time, space cooling and heating units had to be rented to maintain tenant comfort.

Building cybersecurity is a “real thing,” and a serious issue for many properties. However, not all events are caused by malicious intent. A lack of proper policy or IT understanding of these issues can cause enormous problems as well. Join me next week as I discuss the issues an improper cybersecurity rollout can cause.

 

Want to learn more about cybersecurity? Other case studies on scans and compromised printers and connectivity loss are available on CRE Insight Journal. You can also watch experts Fred Gordy and Michael MacMahon present about the cybersecurity framework developed by Building Cyber Security (BCS). Watch now on CRE Insight 365!

 To stay up to date on news and resources such as this and other topics of importance to the real estate industry, subscribe to the free CRE Insight Journal Newsletter using this link.