Technology Today

2012 Issue 2

Monitoring and Managing Cybersecurity Events in Complex Systems: A Multi-Dimensional Approach

Monitoring and Managing Cybersecurity Events in Complex Systems: A Multi-Dimensional Approach

The increasing frequency, rising costs, and growing sophistication of cybersecurity (CS) attacks on Department of Defense (DoD), agency and commercial enterprise systems are dramatically reducing the quality of end-user services and compromising mission effectiveness. Organizations that manage these complex enterprise data systems must simultaneously pursue three goals to minimize the adverse effects of a cyber attack. In order of precedence:

  • Maintain system operational availability and integrity.
  • Fulfill service-level agreements.
  • Protect the systems and the data they carry against cyberattacks, including recognition of the onset of a cyberattack in a timely manner and with cost-effective, threat-appropriate (cyber) protection and mission or policy-appropriate responses.
Figure 1

Current attempts to overcome CS threats include intrusion detection and prevention systems (IDS/IPS), firewalls and packet scanning software.1,2 Individually, these approaches are challenged to prevent or provide sufficient countermeasures to overcome and resolve the wide spectrum of CS threats that affect the multiple and diverse components that compose a complex enterprise system.

To meet this need, Raytheon has proposed a new multidimensional CS approach to monitor, manage and respond to CS events. This approach extends Raytheon's End-to-End Enterprise Monitoring (E2E-EM) Reference Architecture,3 developed under the guidance of the Defense Information Systems Agency (DISA), to include a new response dimension that activates countermeasures to prevent or limit the effects of cyberattacks. This new multidimensional framework (Figure 1), called Enterprise Monitoring and Management Response Architecture for Cybersecurity (EMMRA CS),4 includes: the X dimension (measurement time intervals), which defines time-based events; the Y dimension (domains), which addresses events that are detected and responded to using similar techniques and instrumentation; and the Z dimension (enterprise perspective planes), which introduces structures within which particular end-to-end events are monitored and managed, possibly spanning multiple time intervals and domains. The Cybersecurity Plane, highlighted in red (Figure 1), enables end-to-end, enterprise-wide detection and response for CS events, by applying CS metrics, correlation and countermeasure response.

Figure 2

EMMRA CS is realized using CS agents that are deployed throughout the enterprise hardware and software components (including IDS, IPS and firewalls), where they continuously monitor the enterprise system for CS metrics. Distributed agents have previously been used to combat cyberattacks, such as Distributed Denial of Service (DDoS).5,6 However, EMMRA CS adds a new collaboration capability called collection and analysis nodes (CANs), which correlate the CS metrics information collected by CS agents to provide an end-to-end picture across multiple, diverse administrative domains for security responders. The CANs are distributed throughout the enterprise in various operations (Ops) centers, according to the domains of the EMMRA CS architecture, such that the missions they support determine the CS agents they use and the metrics they analyze. For example, Figure 2 shows a CAN located at the Local Ops Center that could collect/analyze infrastructure domain CS metrics from distributed EMMRA CS agents. Both the collected metrics and the analysis of these metrics would then be stored in the Local Ops Center database that could be exposed over the enterprise system to authorized subscribers, such as security responders and other Ops centers. An authorized Regional Ops Center could then subscribe to the published data from the Local Ops Center's CAN database and collect/analyze additional applications domain CS metrics from distributed EMMRA CS agents. Likewise, an authorized Enterprise Ops Center could subscribe to published data from Local and Regional Ops Centers' CAN databases and collect/analyze additional business/collaboration domain CS metrics from distributed EMMRA CS agents; and an authorized Global Ops Center could subscribe to published data from Enterprise, Regional and Local Ops Centers' CAN databases and collect/analyze additional governance domain CS metrics from distributed EMMRA CS agents. By distributing collection and analysis responsibility, data processing speed and storage requirements are minimized at each CAN, and operator and analyst work load efficiencies may be realized across the mission.

Table 1

The CS agents and CANs communicate over an out-of-band (OOB) network, such as the wireless mesh network shown in Figure 2; therefore they do not impact transport latency or bandwidth over the production network. The integrity of the OOB data is maintained using continual asset discovery to ensure comprehensive agent deployment throughout the enterprise system.

In addition to segregating metrics according to EMMRA CS domain, the network also categorizes the metrics within those domains so that the individuals responsible for resolving the CS threat may be identified (Table 1).7

Figure 3 shows a representative enterprise system that includes strategic network components and interconnection points where EMMRA CS agents can effectively observe the relevant, trusted and high-value CS metrics of Table 1, and then report them to CANs distributed within local, regional, enterprise and global Ops centers. In this example, there are three administrative domains represented by the three circles: Help Desk, Operations and Engineering. End users also publish and subscribe within the enterprise system. Figure 3
The network edge (small circle) comprises client workstations and a Customer Edge (CE) router attached to a High Assurance Internet Protocol Encryptor (HAIPE) through which data are passed to a Provider Edge (PE) router and stored in cache. In the core network (center circle) a Multiprotocol Label Switched (MPLS) optical cloud, Dense Wavelength Division Multiplexing (DWDM) and Synchronous Optical Network (SONET) transport information. The network terminates at a Defense Enterprise Computing Center (DECC) with the connection from a second PE router to a HAIPE device, then to a DECC Edge router (DE) and a DECC LAN. The large circle contains computing services and components, including those for Dynamic Host Configuration Protocol (DHCP), Distributed Names System (DNS), Discovery, Cluster, Apps Server, Portal, Message Queue, Web Server, Service Node, Security/Single-Sign-On (SSO) and Database.

EMMRA CS Verification

A large-scale simulation that emulated the EMMRA CS architecture of Figure 1 was performed in collaboration with Queen Mary University of London, on the U.K. national supercomputing service, HECToR.8 Running this simulation over the network topology shown in Figure 4 verified that the EMMRA CS inter-domain and inter-plane information sharing between CANs enables detection of CS events originating in one plane or domain anywhere within the enterprise system. For example, the simulation was run for Voice over Internet Protocol (VoIP) scenarios in which the enterprise system was subjected to Denial of Service (DoS) security threats. For these scenarios, DoS events detected in the Cybersecurity Plane resulted in a loss of network connectivity.

Those detected in the Control Plane caused the reconfiguration of physical network links. Those detected in the Usage Plane caused degradation of voice service, and those detected in the Management Plane initiated bandwidth re-provisioning. Each of these security events was stored in the CAN database for later analysis, such as total cost impact due to the threat and cost savings based on the EMMRA CS response. Note that the CAN database is not format sensitive, and therefore will not limit the compatibility of appliances and other security tools with the EMMRA CS framework. EMMRA CS provides operators with visibility across traditional boundaries that enable them to proactively respond to security events before they compromise the enterprise, including the attack scenario of a rogue user from within or outside their administrative domains.

Figure 4

EMMRA CS Realization

A unique contribution of EMMRA CS is its ability to leverage new and existing tools and methodologies for application to cybersecurity. For example, EMMRA CS adapted a methodology that was previously used to evaluate Quality of Service (QoS) mechanisms to support VoIP services based on a voice delay metric, called the R-factor,9 to create a novel "whole-of-network" visualization capability from which CS agents can detect and respond to a DoS attack. This new software tool, called the Real-Time Monitoring and Response Tool (RTMRT), supports both manual and automated observation, interpretation and response capabilities for real-time applications. The manual feature is used for addressing unique problem symptoms that have not previously occurred and for which no automated response has yet been determined. In this case, the tool would permit the network operator to manually click on a real-time contour, representing the metric category of interest, and then "push down" into the next level of monitoring for that metric category.

At this next level, the structure of the EMMRA CS framework and reference architecture directs the network operator to the most likely place in the enterprise system where the CS event could have occurred. At this monitoring level, specific CS metrics names are applied to permit the operator to quickly distinguish the exact problem; thereby enabling faster problem resolution and QoS restoration.

The tool automates the manual process for problem symptoms with known resolution approaches, thereby permitting faster response and resolution for time-critical issues.Now that RTMRT software has been tested in a simulated environment, the next step is to deploy this software in an operational environment. Instances of the EMRMA CS hardware agents and CANs have been designed for and deployed on operational and experimental networks that transport data over fiber optic connections at rates of up to 40 gigabits per second.10


Through its unique approach to cybersecurity event monitoring, management and response, EMMRA CS addresses the key enterprise systems service provider goal of protecting their systems and the data they carry against cyberattacks. EMMRA CS enables operators and analysts to recognize the onset of cyberattacks in a timely manner by proactively identifying the threat(s) and recommending response countermeasures to thwart them. Based on the distributed architecture approach, EMMRA CS is scalable and it enables cost avoidance by not requiring all capabilities to be implemented within every operations center. It provides a cybersecurity solution that applies to DoD, agency and commercial complex enterprise systems.

1. Defense Information Security Agency, "Network Infrastructure Technology Overview." Version 8, Release 5, 27 April 2012.
2. Defense Information Security Agency, "Enclave Security Technical Implementation Guide," Ver. 4, Rel. 3, Jan. 2011.
3. P. Hershey, D. Runyon, and Y, Wang, "End-To-End Enterprise Monitoring Framework for NetOps," Proc. of MILCOM 2006, Washington, DC, Oct. 25, 2006, pp. 1-7.
4. P. Hershey and C. Silio, "Procedure for detection of and response to distributed denial of service cyber attacks on complex enterprise systems," in 6th Annu. IEEE Intl. Systems Conf. (IEEE SysCon 2012), Mar. 2012, pp. 85–90, doi:10.1109/SysCon.2012.6189438.
5. A. Kumar and S, Selvakumar, "Distributed Denial of Service (DDoS) Threat in Collaborative Environment – A Survey on DDoS Attack Tools and Traceback Mechanisms," Proc. of 1st Intl. Advance Computing Conf., Chennai, India, Mar. 2009, pp. 1275-1280.
6. U. Akyazi and A. Uyar, "Distributed Intrusion Detection Using Mobile Agents Against DDoS Attacks," 23rd Intl. Symp. on Computer and Information Sciences, Istanbul, Turkey, Oct. 2008.
7. P. Hershey, D. Runyon, Y. Wang, "Metrics for End-To-End Enterprise Monitoring of Enterprise Systems," Proc. MILCOM 2007, Orlando, FL, Oct. 19, 2007, pp. 1-7.
8. P. Hershey, J. Pitts, and R. Ogilvie, "Monitoring Real-Time Applications Events in Net-Centric Enterprise Systems to Ensure High Quality of Experience," Proc. IEEE MILCOM, Boston, MA, Oct. 2009.
9. J.M. Pitts, J.A. Schormans, "Configuring IP QoS Mechanisms for Graceful Degradation of Real-time Services," Proc. IEEE MILCOM, Washington, DC, Oct. 2006.
10. P. Hershey and C. Silio, "Surmounting Data Overflow Problems in the Collection of Information for Emerging High-Speed Network Systems," IEEE Systems Journal, Vol. 4, No. 2, ISJEB2, June 2010, ISSN 1932-8184, pp 147-155.

Paul C. Hershey

Share This Story

Top of Page