Cybersecurity is a buzzword with which every business and individual with internet access is familiar. “If the machine is not on the internet and uses no CDs or flash discs, you’re safe,” says Mauritz Botha, co-founder and chief technology officer, XiO. However, he points out, an isolated system is inconvenient.
In a connected world, cyberattacks are pretty much inevitable; they are going to happen at some point. According to Forbes, the Online Trust Alliance tracked more than 159,000 cyber “incidents” in 2017. A report by Positive Technologies reveals that cyberattacks increased 32% from the first quarter of 2017 to the first quarter of 2018. The report goes on to state that 63% of the attacks targeted data and credentials. Malware attacks intended to obtain information including account logins, Social Security numbers, and security question responses increased 75%
in 2017.
ON THE ATTACK
However, too often, companies, individuals, and utilities are not proactive. “People need to accept that attacks will occur and that some are going to be successful,” warns Michael Kanellos, industry champion, water, OSIsoft. “It’s an attitudinal shift, but it takes people away from looking for a golden bullet or an ultimate solution to thinking of security as a collection of technologies, judgment calls, and worst-case-scenario planning.”
Forbes expects the global cybersecurity market to reach 170 billion by 2020, with companies spending as much as $96 billion on cybersecurity in 2017—most of that in reaction to breaches instead of prevention of them.
More people are trying to get into the system than ever, observes Jason Bethke, president and chief growth officer of FATHOM Water Management. It’s due, in part, to the fact that utilities collect more sensitive data, such as personal information for billing, than ever.
That data is subject to cyberthreats in the form of application attacks, malware, ransomware, phishing, and exploit kits. Sophisticated and even automated attacks can disrupt the infrastructural assets of a wastewater treatment plant, as well as its customers. Cyberthreats can take many forms:
- Cyberterrorism attacks networks, computer systems, and telecommunication infrastructure.
- Cyberwarfare penetrates networks to cause damage or disruption. Instead of shutting down a network, it typically compromises data, degrades communication, impairs services, and/or interrupts commerce.
- Cyberespionage uses technology to obtain information without permission.
These threats are enabled by rapidly advancing technology with ever-evolving security requirements, including “bringing your own device, the Internet of Things, cloud-based applications, and remote access.”
To combat these threats, cybersecurity incorporates a set of methods, systems, and plans to protect networks, programs, and data from unauthorized access and to protect computer systems from theft or damage to hardware, software, and electronic data, as well as cyberterrorism (the disruption or misdirection of services). It has become big business.
McAfee and The Center for Strategic and International Studies says these breaches cost companies $608 billion. Clearly, it’s an important issue and is best dealt with by taking proactive measures.
TWO TIMES THE PROTECTION
Utilities have two networks: the IT network that controls things like the billing system, email, vendor orders, and the like and the OT network that controls equipment like valves and pumps and collects critical data on metrics like water level, pH, flow, and water quality. “The OT (or operational technology) network is really the nerve center of a utility,” believes Kanellos.
He advocates separating the two networks, or at least insulating the OT network from the IT one, because most viruses and ransomware attacks come from the IT side. “That’s changing, but it’s still where most come from. The type and number of devices on IT networks also changes more rapidly, creating the potential for exploitation.”
They can be isolated by a complete air gap with no connection whatsoever. However, that creates other challenges because business planners and others can’t get to live operational data to see if there’s a problem.
One solution Kanellos suggests is a one-way diode. “Basically, data can leave the OT network, but data can’t come in,” he explains. Waterfall Security, among others, makes these. They are impervious, but they cost around $30,000.
Another technique is a video feed. This has been used by Maynilad, the Philippine water agency for Manila serving around 9 million people. The OT screens are transmitted by video to the IT analysts and others. They can see what’s happening, but they can’t interact with the data like they could with a diode. “It’s a clever solution if you just want to give more people access to the data,” sums up Kanellos.
Those ideas work for preventing attacks from the IT side. Solutions for attacks directly on the OT side are starting to emerge. Power utilities are finding rogue code on their OT networks, Kanellos reveals, adding that there is increasing evidence that these attacks could be used to control equipment.
The best defense is to exercise eternal vigilance and practice. Keep equipment updated. Conduct periodic security checks. “Also,” he adds, “it’s best to come up with a plan on what to do after an attack occurs. It’s inevitable: think of this as a fire drill. Who calls the regulators? When do you let the public now? If a reservoir gets knocked out, what would be the alternative reservoir? The idea is to resolve the problem in the least amount of time. Practice helps.”
DISASTER RECOVERY
In addition to focusing on security, Bethke says it’s imperative to think about disaster recovery and redundancy. “It’s a people-process-technology issue. You need good monitoring and alerting. You should be running vulnerability scans. You must have redundancy and training to promote security awareness (such as rules for credit cards and data security). Good end-user security requires updates, passwords, and the right people doing the monitoring.”
He says those people need to monitor compliance by conducting an audit and annual review of policies. Who has access to data? Is the data encrypted? It’s not a one-time deal, either. As the technology upgrades, the tools to protect it need to evolve.
“It’s a lot for a utility to tackle,” states Bethke. “Utilities are small and fragmented. For them, the cost of a career path to recruit IT talent is high. They look for career progression, but that’s often lacking at a utility. They have no place to go.”
In addition, he says, the tools haven’t reached small IT groups. Alerts, alarms, audits, and security awareness—it’s a lot to manage. “We put all that into a service contract so the utility doesn’t have to build it,” says Bethke. He believes it’s “better to seek services for scale protection.”
Economies of scale can mean that subscribing to a service, not unlike an anti-virus protection service many computer owners use, is a more efficient and effective method of cybersecurity. “A utility doing its own security is suicide,” insists Botha.
“Cybersecurity is a moving target,” he continues, agreeing that there is “a lot of fragmentation” in the utility sector, which covers countless small towns as well as major cities. It can be better to let a third party aggregate the information and monitor it.
Handing off cybersecurity to professionals can be the easiest way for a utility to protect itself. XiO, which has been providing protection “since the 1970s, when there was not even a word for cybersecurity,” hosts data on secure architecture with levels of separation. They believe in compartmentalizing to fight intent. “Not everyone can access everything—and you can’t access some parts unless you’re in the office,” reveals Botha.
What does security mean? Botha proposes several questions to ask a provider, starting with what industry standards do they follow? A provider should run a security audit, offer a confidentiality system, ensure integrity (that the information is not tampered with), possess the ability and availability to recover in a timely manner in the case of a breach, and have a contingency plan—a configuring management plan.
Every six months, XiO starts from scratch with their cloud system to automatically re-deploy all their servers. In addition, they hire consultants to review changes every month. “We don’t even trust ourselves!” jokes Botha.
On a serious note, he says that part of security is disaster recovery. “You need a recovery plan.” A plan for quick recovery needs to be in place…and should be tested. He references Netflix’s “chaos monkey concept” in which someone breaks the system to test the staff’s ability to recover.
Whether that’s the best option for a wastewater treatment plant or not, adding a well-defined process is crucial. Layers, although they complicate the process, add security. Most hackers do not gain all-access, Botha explains. Instead, “some get a toehold, some anchor here and there. It’s a slow penetration before they steal information.”
FATHOM’s secure customer portal
VULNERABILITY
A major data breach occurred when an HVAC vendor accessed Target’s system. Remote access from cell phones can also leave utilities vulnerable to attack. “You must have a well-defined system in place. It should not be an afterthought,” urges Botha.
Customer portals allowing visibility of billing history and the ability to pay bills from a cell phone can open a utility to susceptibility, making encryption of customer data crucial. Operational requirements that allow customers to see their usage are popular, but they aren’t always updated. “The bad guys figure out high-value targets with less effort to get to,” says Bethke. “Without proper monitoring and updates, you’re vulnerable.”
Real-time scams can also jeopardize operation. Botha says XiO picked up a new customer that had been the victim of ransomware—a booming business, he notes—because “everything came to a standstill” until it was settled.
Similarly, Bethke recalls one municipality that had to shut down when their software was ransomwared until a manual work-around was established.
There is also a variant referred to as “machine phishing attacks.” In regular phishing, a hacker poses as a CEO or the IRS in an email, demanding social security numbers. In a machine phishing attack, the equipment might tell an operator, “This valve is not open,” when, in fact, it is open. “The engineers don’t move to close it and problems ensue,” says Kanellos. “We’re working with Lawrence Berkeley National Labs on a program to stop these by comparing network data, machine data, and first principles of physics.”
For example, if the message comes in, “This valve is now closed,” but the machine data—often on a different network—says water is flowing and physics says the valve couldn’t have closed that quickly, something is wrong. “The platform we’re developing would aggregate the relevant data sources in a way that a human could make a quick, reasonable judgment and act,” explains Kanellos. “This system essentially assumes that some sources of data could be bad. The idea is to give you a chance to evaluate all of your data.”
Physical attacks also occur. In 2013, someone cut the fiber optic lines at a power substation in San Jose and then shot out 17 transformers. The utility couldn’t tell the fiber had been cut. Kanellos says that more, independent sensors could help ameliorate the information blackout.
“Could a water utility face a physical attack?” he reflects. “Sure. A lot of reservoirs and pipes are in isolated areas. Sensors or software that monitors things like flow rate can help detect a problem early.” He mentions that Evides in the Netherlands has developed a program that lets them identify the location of major pipe bursts within minutes. “The pipe bursts in all likelihood aren’t security breaches, but if they were, the system would have the same impact: early warning.”
THE HUMAN TOUCH
Water utilities can also plan against downstream security problems unrelated to human interference. For example, the City of Calgary regularly faces the potential of spring flooding because it sits at the confluence of two rivers that can overflow in the thaw. To reduce the risk, Calgary analyzes the snowpack, reservoir levels, expected consumption, melt rates, and other metrics, and then shifts water around to different reservoirs to lower the chance of sudden, uncontrolled flows.
Mother Nature can create problems, but often, security breaches are the result of human interference, whether malevolent or not. In regards to the protection of the operating technology, Stefan Woronka, business development and regional management, Industrial Security Services, Siemens, says they “regularly find that proven concepts and best practices are not implemented properly.” He indicates that lack of dedication and resources is a typical hurdle. “Someone has to be appointed to protect the Industrial Automation and Control Systems. He or she has to be assigned sufficient resources (manpower, time, money) to take care [of] this.”
Siemens uses and endorses the use of the IEC 62443 and its concept of defense in depth. This concept introduces several layers of protection to a critical asset. The layers range from physical protection to the implementation of proper guidelines and procedures, over to network segmentation using a “zones and conduits” model to final system integrity that takes care of protecting the endpoints. The different layers together build a stronger resistance against attacks. How strongly and intensely each of the layers is implemented depends on the business impact and the consequences of attacks to each of the assets. The higher the criticality, the more measures should be implemented.
MASTERING CYBERSECURITY
Master Meter’s data management systems support cloud infrastructure to allow utilities to leverage virtually unlimited scalability and security, according to Vlad Andresscu, product manager, software. “All of our services leverage Microsoft’s Azure cloud, which provides [a] comprehensive approach to [the] remote/IT environment.”
Cyber criminals are automatically denied access, thanks to the system’s active directory that manages and controls the identity of each authorized user via multifactor authentication. Further protection for data in transit and at any storage point is provided by key vault and data encryption via hardware secure modules, secure socket layers, internet protocol security, and advanced encryption standards. In addition, all processes that utilities require to execute daily are based on a site-to-site virtual private network, network security groups, and physical infrastructures security for data disaster recovery.
“Interestingly enough,” says Andresscu, “one of the main challenges is simplifying and outlining the cloud services benefits to water utilities (as opposed to localizing data on premise).” Cloud services aren’t about being the “latest technology,” but rather are a natural progression of scientific advancement that is impacting most trades (not just water industry). “It allows companies of any size to excel at their business, streamline their products, and minimize their on-premise assets [such as] localized servers, IT support/management, etc.”
LEVERAGING TECHNOLOGY FOR THE FUTURE
Improving cybersecurity for increasing and evolving threats is a challenge. First, Botha cautions, you must understand how the system works, including artificial intelligence and big data. “Technology can be used to learn about the system. That’s significant because there will be a big departure of skilled labor, so we need to capture institutional knowledge before they retire.”
One hot topic that is driving change is Industrial Anomaly Detection (IAD). This solution solves several issues. IAD helps with the identification and classification of assets in an Industrial Automation and Control System. Often, the asset inventory was accurate on the day of the system going live, but has not been updated to accommodate changes during its life cycle.
IAD maps all existing communication and thus creates a communication matrix. Based on that and an updated inventory, it’s possible to monitor for abnormal or unwanted behavior—hence, the name Industrial Anomaly Detection.
Siemens has released an offering for Industrial Anomaly Detection that helps to solve these three issues. Woronka explains that the solution will be connected to the Automation System through mirror ports or test access points (TAPs) to ensure minimal to no impact to the Automation System. The traffic of the Automation System will be copied to the IAD in order to identify, classify, and monitor the assets.
While FATHOM’s Bethke considers the water industry just at the beginning of adoption—the infrastructure phase designed to “get good data flow”—he already envisions the next evolution creating proactive processes to be more effective. “Now we want to figure out how to expand data to lake and well customers.”
He also foresees leveraging infrastructure for smart cities platforms that could incorporate things like lighting. “The technology is getting better,” says Bethke. “The smart grid is the starting point for small cities.”
Becoming “gentler” on that infrastructure, rather than having to replace it all, is also important, as is proving that efficiency can pay for technology. Bethke predicts the introduction of business models that utilities can adopt at incremental costs. Those models involve outsourcing cybersecurity. “Not enough [water utilities] transfer risk from the city, so there has been slow adoption.” But service contracts transfer risk. “The customer is not buying the product; they’re buying the result.” It’s up to the cybersecurity provider to provide that result, which he believes can be done more efficiently and economically than if the water utility attempts to do it on their own.
However wastewater treatment plants choose to address cybersecurity, it’s clear that data breaches and ransomware can have a long-lasting detrimental effect that costs them a substantial sum and can compromise customer data. Taking measures to prevent cyberattacks instead of reacting to them can ultimately save valuable time and money.
