A chain of InfoSec episodes that took place in the time span of two years and concerned the same company. What practical conclusions can be drawn from these episodes? Let’s start with a brief analysis of what happened, identify probable causes, and possible ways to prevent it.
#1. Who’s to blame?
Defining the root cause of events is one of the main quests. Can using SoHo equipment in a production facility explain everything? Or does the problem arise from the integration of multiple products and the blurred responsibility between dozens of contractors?
A national-scale distributed transportation company, with all the resources it manages, is a benchmark example of a complex system. Not just any complex system at that, but a VERY COMPLEX SYSTEM. Efficient operation is impossible without process automation and digitization. Otherwise, every increase in the number of system components would require a nonlinear growth in personnel numbers. And that would make the point of failure shift from technology to a more unpredictable field - the human factor.
We have to give some credit to the aforementioned company: they have been quite successful in automating their business processes. However, automating an extremely complex production results in the creation of enormous IT systems, that inherit the properties of the original EXTREMELY COMPLEX SYSTEM. It’s not the depth of the technological solution stack that poses the challenge, but the sheer number of entities and components comprising the system. When the number of entities is measured in thousands, the manageability of the system and its operation transparency is lost. While the number of different events and control points grows rapidly, the amount of mutual influence possibilities is that number squared.
It should not be assumed that only the architects of the solution, who chose to implement cheaper equipment, or the contractors, who failed proper implementation and commissioning, are at fault. Typical project management mistakes such as cutting project budgets on the go, do not sound like a sufficient explanation either.
Truth is, in recent years, the company has experienced a qualitative leap in system complexity, that exceeded the threshold marking the efficiency of the standard management methods. The operator has lost control over the manageability of the system.
#2: Solving the puzzle
Two possible solutions exist for a situation when system complexity increases rapidly:
1. One option is to perform an administrative breakdown of the entire project into separate sections, delegating control to the dedicated groups of operators. This will automatically lead to an increase in the number of specialists involved (and this increase will be non-linear), which, in the case of a very large system, can in turn boost the company's operational expenses effectively nullifying the initial economic benefit of the automation. This approach is somewhat reminiscent of the «South Park» episode, in which the town's population became so keen on ordering goods from Amazon that as a result, everyone had to work at Amazon's logistics center. Lack of qualified personnel poses another challenge here.
2. Using automation tools to aid IT/InfoSec specialists is a rather viable alternative. These tools can provide visibility into system operation and allow monitoring of the significant parameters for large and geographically distributed systems, while keeping personnel requirements sensible.
As the result, the company may either build a system that would feature a person, responsible for each node or segment operation, and adopt the corresponding staff expenses, or invest in automating the process of collecting, processing, and analyzing the security events generated by the system.
#3. What about the risks?
This example actually features a Critical Information Infrastructure (CII) facility, and in most countries, CII operators are bound to comply with quite a few regulative requirements imposed by the authorities. The fact of public disclosure of the so-called audit results might result in increased attention from various agencies, which may include extraordinary compliance audits and other activities. And it gets worse when consequences are not limited to direct financial losses in form of fines, but also include reputational losses, and, for public companies, ultimately lead to negative capitalization changes.
Let’s review the first incident considering the worst-case scenario. The researcher managed to obtain access to the CCTV system, which means that an adversary could potentially carry out a direct attack on the technical security system, or impact CCTV operation in one or more public places. Station information boards control had also been among the compromised systems, meaning that the intruders might display any sort of messages to an unlimited number of people. These actions may not look hazardous for the primary business processes of the company and general safety, unless we consider them as the first steps of another, possibly more extensive and malicious attack.
#4. Automation options for the workflow of an InfoSec specialist?
The work of an information security officer is not limited to setting up and maintaining asset security at the required level, but also includes a significant amount of paper processes, interaction with internal and external authorities, maintaining operation logs and protocols along with multiple other documents. These clerical tasks require as much labor as initial system design and implementation and are a pain point for many. But what if the bulk of it is switched to automated tools?
The market offers specialized solutions that allow companies to streamline the process of monitoring and controlling the state of security in automated systems. As noted earlier, maintaining the necessary InfoSec state should not rely on a stack of various security products and consoles, effectively making the initial problem worse. As a developer of InfoSec state analysis and monitoring tools for the OT, IoT, and IIoT environments we embrace the approach of providing a single operator console that would encompass all necessary functions for providing a comprehensive view of the protected system. Several key areas can be identified:
Network traffic analysis.
An in-depth analysis of the network traffic is essential for discovering the assets, determining system topology, and monitoring the interactions that exist between the nodes. Concerning the cases, studied above, this mechanism could have provided the tools necessary to detect the anomalous network activity that had not corresponded with a normal system operation profile, i.e. the connections between the nodes that had been initiated by the enthusiast researcher and should not have existed during regular operation.
Configuration monitoring.
Collecting configuration files and comparing them with the reference allows precise detection of the unauthorized changes in asset configurations, including hardware, software, and user lists, changes in security settings, services activation/deactivation, and many other areas. In our cases, monitoring the configurations could have enabled the operator to track changes in the assets’ network settings that had been introduced by the researcher in order to gain remote access.
Event analysis.
In most cases, protected assets are able to notify the operators when InfoSec-related events occur. The key here is processing the flow of notifications in a way that allows the important events, that may indicate an ongoing InfoSec incident, to be displayed on the dashboards immediately, while filtering out insignificant events, that may overload the operator or clutter the view. A perfect example here would be a single unsuccessful login attempt versus multiple attempts in a row — the second case is quite likely evidence of a password brute-force attack.
InfoSec compliance control.
Assessing the compliance status of the assets that belong to the protected system with corporate, national, or industry requirements is an essential step to bringing the whole system into compliance. Automating the assessment process relieves the personnel from routine operations allowing human operators to focus on important tasks. In our cases, this assessment could have alerted the operators that a proxy service had been enabled by default on a certain network node.
Vulnerability Analysis. Identification and analysis of known vulnerabilities that exist in the system aids in proper and timely mitigation before those vulnerabilities could be exploited by an adversary. Regarding our case, a vulnerability analysis performed regularly could provide operators with a list of devices running outdated firmware.
Naturally, implementing an InfoSec analysis and monitoring tool implies direct and indirect costs, however, the focus should be on the most significant risk that InfoSec vulnerabilities pose – business processes interruption. Distributed networks and IT systems are built to facilitate the business processes of the company, but how far is an unaddressed vulnerability, that allows external parties to get access to the system with minimum effort, from direct and possibly malicious interference with the core business?
A simple attack example, relevant for logistic operations, would be compromising the cargo tracking database. How would a company assess and clear the chaos in shipments planning and execution? How would it remediate the damage and losses that result from contract terms violation and subsequent claims?
Summary
Finally, let’s pay some attention to the elephant in the room — IT systems, particularly networks, quite often are not just tools for business processes’ support but also facilitate the operation of other systems, that control technological processes. There are known cases when successful ransomware attacks, brought the affected industrial control systems to an inoperable state within a few hours, resulting in downtime for large industrial enterprises, as the technological processes were overly sophisticated for any sort of manual control. Naturally, each unplanned downtime for a large production facility resulted in significant (millions of US dollars) losses for its owner.
The topic of securing the OT infrastructure and the need for companies to comply with all the different rules and regulations for CII owners is nothing new, however reflecting on cases similar to the ones described above proves that it is as relevant and important as ever, especially as digital transformation takes place across multiple industries.
The scope of companies that could benefit from the automation of the InfoSec management processes is not limited to large distributed enterprises but encompasses all businesses that operate complex systems, the security of which are a key factor not only for company success but possibly for the safety of human life.
To sum this up, let’s refer to a famous quote [4] of Bruce Schneier:
Security is a process, not a product.
And remember that processes in both large and complex systems require modern automation tools unless the world is willing to see similar cases, concerning other enterprises.
No matter how experienced and skilled may the employees responsible for system security be, their capabilities are limited by human nature itself: the amount of information they can process, reaction speed, the effects of fatigue, and vision upset along with other factors. Humans have long learned to significantly expand the boundaries of their capabilities through the use of various tools, and it is about time that InfoSec officers obtain and adopt the tools, adequate to the complexity of the tasks they face.
References
[1] https://habr.com/ru/post/536750/
[2] https://habr.com/ru/post/476034/
[3] https://www.interfax.ru/russia/686957
[4] https://www.schneier.com/essays/archives/2000/04/the_process_of_secur.html
[5] https://gudok.ru/newspaper/?ID=1552569