HPC Leading Edge News
HPC Leading Edge Resources
May 24, 2017
By Rob Farber, global technology consultant, TechEnablement
There is always a tension between collaboration and cybersecurity: Open collaboration raises fears that the system security will be compromised, yet locking a system down hampers research and ultimately interferes with the reasons for providing computing resources.
Jimmy Pike, a Dell Fellow, summarized the balance between cybersecurity and collaboration as:
1. know who you are talking with, and
2. preserve your ability to trust the user ID associated with that individual.
In cybersecurity terms, that means vetting individuals before granting access identification, and then following policies to ensure the known access ID remains a trusted quantity.
“With the state of the world, we need to collaborate, yet any public collaboration is suspect,” Pike says.
There are some excellent tools out there for collaboration, ranging from free offerings like wikis, Google Docs and GitHub to purpose-built software. Interactive tools are readily available for sharing screens, voice and data. The challenge with balancing collaboration and cybersecurity always rests with the human in the loop, which means that it is difficult to ensure continued trust. This is where the heart of the cybersecurity problem lies, and even low-technology approaches, such as email phishing attacks, can compromise a user.
To combat those types of attacks, Pike advises organizations to “reduce their attack surface,” which is where the technology innovation is happening.
NERSC (National Energy Research Scientific Computing Center) has eliminated a site-wide firewall. Instead, it uses resource-specific firewalls to protect staff and core internal resources. With a mission requirement to enable an open and collaborative high performance computing environment, the NERSC cybersecurity work is paradigm-changing for cloud and traditional IT infrastructure cybersecurity staff.
NERSC is a high-profile security target that supports nearly 6,000 Office of Science users, so the NERSC staff has to perform a balancing act between enforcing cybersecurity protocols and providing open accessibility to thousands of technically literate, high performance computing (HPC) scientists. The NERSC Cray User Group (CUG) paper, Experiences with Intrusion Detection in High Performance Computing states: “We have come to see the role of cybersecurity as an enabling technology to facilitate maximum performance and functionality.”
IT staff who work with site-wide firewalls know that these firewalls simplify security by mandating a single access point—thus dramatically reducing an organization’s “attack surface.” Unfortunately, as most big data researchers know, the protection that a site-wide firewall provides occurs at the expense of imposing a (potentially debilitating) data bottleneck, as all network traffic must traverse that single resource. Moving many gigabytes of data can be a challenge, and moving a terabyte generally requires using the bandwidth transfer capabilities of a FedEx truck and a storage device.
Low-Impact Distributed Site-Wide Monitoring
Without the simplification of a site-wide firewall, the NERSC security team assumes from the beginning that an attacker might have already landed on a system. It also means that high-speed big data collaborations can occur without forcing all site-wide traffic through a single device.
Instead, NERSC enforces cybersecurity through distributed monitoring to preserve performance and keystroke monitoring through a modified secured socket shell (SSH) daemon.
The wide adoption of SSH as the standard form of communications between users and HPC services (at NERSC and other sites) has successfully avoided traditional key logging and man-in-the-middle attacks. However, these encrypted SSH connections also create problems for the security groups tasked with monitoring network-based logins and activity.
To facilitate the monitoring of user connections, NERSC runs an instrumented SSHD (iSSHD) on their servers. The instrumentation provides application layer data like keystrokes and login details, as well as metadata from SSHD such as session and channel creation details.
The Bro Monitoring System
NERSC also uses the Bro Intrusion Detection System (IDS) to filter and normalize data. The Bro IDS can provide deep (and normally expensive) analysis of high-volume traffic without significant packet loss or exceptional expense. For more on the Bro IDS, please see the Vallentin et.al NIDS Clustering paper.
The approach taken by the Bro IDS and organizations that eliminate the site-wide firewall reflects a focus on looking for anomalies rather than attack signatures. The cybersecurity team looks for behavior outside the norm for their user base.
The anomaly-based approach is proactive in that suspect users can be identified during an attack. The cybersecurity team can contact the user for confirmation and re-establishment of trust. The user ID can be restricted, locked down or deflected to a honeypot. In contrast, many signature-based approaches assume trust until postmortem signatures are found.
The challenge boils down to the people on the cybersecurity staff. Distributed monitoring, the elimination of a site-wide firewall bottleneck and use of tools like the Bro IDS means the cybersecurity team needs to understand common protocols like HTTP, SMTP and DNS. The support network for the IDS is key to selecting the right tool, as going it alone is not viable from a security standpoint, nor a recipe for success.
Email phishing scams have historically proven to be excellent ways for hackers to assume an identity from which they can then break into other systems. The recent spate of real estate email scams represents just one example. Basically, criminals hack into the email accounts of real estate agents or others involved in a real estate transaction to use the information to dupe a party into a fraudulent wire transfer. Some people have lost their homes. Similarly, a government lab was taken offline several years ago by a phishing email that installed what is referred to as an “advanced persistent threat.”
Pike notes that we are just entering the Internet of Things (IoT) era, where mass numbers of internet-connected devices send information in from the edge.
“What happens when malicious devices slip in?” he asks. Meanwhile, the human in the loop does not change: “People just want it to work.”
In other words, IT technology experience or capability is not and cannot be assumed by vendors for the users of their products. This is the conundrum of edge and collaborative computing vs. cybersecurity.
Regardless of the technology or approach (signature vs. anomaly), preserving the trust in a user ID is the main issue and the weak point of any set of cybersecurity policies. Examples abound where very low-tech email approaches (e.g., making an official-looking email) can circumvent the most sophisticated security measures simply by compromising trusted users.
After that, technology can be used to provide a secure yet collaborative environment. In particular, giving users sufficient bandwidth for collaboration is key. However, this is a treadmill, as organizations must adapt as data sizes increase. At some point, people have to reconsider imposing a site-wide firewall.
Questions to Consider
NERSC is one of the flagship organizations that have shown a high-bandwidth distributed cybersecurity model can enable high performance and collaboration. Organizations can revisit their current thinking based on the success of the NERSC approach and ask:
“Can we eliminate our site-wide firewall without compromising security?”
“Should we consider adopting an anomaly-based cybersecurity approach?”
This second question is obviously less disruptive to the current state of how business is done, and both signature and anomaly-based approaches can be used concurrently.
For more info:
Rob Farber is a global technology consultant and author with an extensive background in HPC and in developing machine learning technology that he applies at national labs and commercial organizations. He can be reached at [email protected].