Why Your Enterprise Security Controls Break on an HPC Cluster (and What Replaces Them)

Security wants an agent on every node. The HPC team says no. Six months later the ticket is still open, and the reason it has not moved is that both of them are right.

The security architect is reading a control catalog that says every host runs endpoint detection, and they are not wrong: that is what the baseline says, that is what will be asked for by name, and pointing at a supercomputer does not make the question go away. The HPC director is protecting a machine whose acceptance test was a benchmark, bought by people who will hold it to that benchmark for years, and they are not wrong either: the agent takes a measurable share of the thing the machine exists to deliver. Nobody in this argument is being careless. The argument does not resolve because it is being held at the wrong altitude, as a yes or no on one control, rather than as a translation problem with a technical answer.

So it ends the way these end: an exception, a paragraph of justification, and a line in the risk register that absorbs the whole thing. The register is now holding a translation problem, which it cannot solve and will carry indefinitely.

The three assumptions

Your controls assume a host you administer, software you approved, and a machine you can reboot. A cluster gives you none of the three.

That is not a complaint about the catalog. Those assumptions are load-bearing for almost every control statement in general use, and true of nearly everything else in the estate, which is why nobody writes them down. A compute node fails all three at once, and it fails them structurally rather than through neglect. It is provisioned from an image and handed back to the scheduler when the job ends, so administering it individually is meaningless. The software that matters on it was written by a researcher, built on the machine that morning, and run by the person who wrote it. And it may be holding one rank of a job that waited a week in the queue, so the reboot you want is not a maintenance action, it is a data-loss event for somebody's science.

None of that makes a cluster unsecurable. It makes a specific control statement wrong, a smaller and far more tractable problem. Below are ten controls a security team will ask for and an HPC team will refuse, in two groups, because they fail for two different reasons.

Part one: the controls that work exactly as designed, and cost you the machine

These five are not misconfigured and not badly implemented. They do the job they were specified to do, and the bill arrives as throughput: latency on the interconnect, jitter across a synchronized job, traffic on a network built for something else.

The EDR agent on every host

The control assumes a host with spare capacity, doing work that belongs to it. A compute node is neither. A tightly-coupled parallel job is barrier-synchronized: every rank waits at each step for the slowest one, so a delay on any single node becomes a delay across all of them, and an agent that wakes on its own cadence produces exactly that delay. Petrini, Kerbyson and Pakin's "The Case of the Missing Supercomputer Performance" (SC 2003), an early and still-cited measured result, traced a large machine's shortfall against its own predicted performance to periodic background activity on individual nodes, amplified by synchronization at scale. Then there is the arithmetic: license and telemetry both multiply by node count, and the telemetry lands on your network. What replaces it is eBPF and sampled instrumentation, out-of-band collection, and telemetry drawn from the fabric and the scheduler, which already know which account ran what, where, and for how long.

Inline firewall, IPS and microsegmentation

The control assumes traffic passes a point where a device can see it. On the interconnect it does not. RDMA lets one node's adapter write directly into another node's memory without the kernel at either end taking part, so there is no host stack to tap and no packet to hand to an inline appliance. That leaves the appliance in one of two positions: off the path, faithfully inspecting the traffic that happens to route through it while the interesting traffic goes around, or on the path, reimposing the microseconds the fabric was bought to remove. NIST SP 800-223 records the same tension at the edge of the machine, noting that a ScienceDMZ architecture is often used for data transfer nodes instead of a pass-through firewall because of the firewall's performance impact. What replaces it is segmentation designed into the topology and the scheduler, enforced at the zone edge rather than in the hot path.

Encrypt everywhere, inspect everything

The control assumes the data path has cycles to spare. Strong scaling is mostly a story about I/O: as you add nodes to a fixed problem the compute time falls and the time spent reading and writing does not, until the filesystem is the entire run. Per-operation cryptography and inline content inspection tax that exact path, at the point where the machine is already at its most fragile, and they do it on every operation rather than on the ones anyone cares about. The replacement keeps the property and moves the cost. Encrypt at the controller and on self-encrypting media, below the filesystem, where the work is done in hardware. Scope access to the project rather than the individual. Turn on root-squash so a compromised client cannot walk an export as root. Then design the staging path deliberately: what lands on scratch, what leaves, and by which route.

Per-host SIEM forwarding

The control assumes hosts are individually interesting and that the number of them is something you write down once a year. Multiply an ordinary per-host log volume by a few thousand identical nodes and three things break together: the ingestion pipeline, the network the logs ship over, and the license you are billed under. The deeper problem is that the record is the wrong shape even when it survives. On a cluster the accountable unit is the job, not the host: a node runs one project's work and then another's within the hour, and a per-host line does not say which. What replaces it is sampling and aggregation at the fabric and the scheduler, shipped out of band, keyed to the job and the account rather than the hostname. NIST SP 800-234 treats this as a problem in its own right: HPC logging is one of the fourteen groups its tailored controls are sorted into.

Authenticated vulnerability scanning

The control assumes hosts drift apart. Compute nodes do the opposite. They are provisioned from a single image and are identical by construction, and in the diskless variant NIST SP 800-223 describes, the image is the only thing a node has ever run. Authenticated scanning across thousands of them re-answers one question thousands of times, and the traffic to do it lands on the interconnect. What replaces it inverts the order. Scan the image, once, where the answer is authoritative and cheap to produce. Attest the node at boot so you know which image it came up on. Then detect drift from that image, because a node that no longer matches what it was built from is a better finding than a vulnerability list you already had. The evidence you hand an assessor is the image scan plus the attestation record, which is a stronger claim than a per-host scan that was true at three in the morning.

Part two: the controls that assume an operating model you do not have

These five cost nothing in performance. Not a microsecond. They fail on the operating model underneath, on who administers the host, who installs the software, and when you are allowed to reboot, which makes them harder to argue about, because the failure never shows up in a benchmark. It shows up as a control marked implemented that describes something nobody is doing.

MFA and SSO on every login

The control assumes one identity per person arriving through one front door your identity provider owns. A cluster has hundreds of users on shared login nodes and one filesystem, arriving over SSH from institutions you do not employ anyone at, on accounts that frequently outlive the grant that created them. And the credential is usually a static key: SSH keys have historically traveled freely between sites with the people who hold them, which turns one stolen key into access at more than one facility. What replaces it is short-lived SSH certificates instead of keys that never expire, federated research identity so the home institution does the authenticating, and multi-factor enforced at the access zone, which exists for precisely this. NIST SP 800-234 gives identification and authentication, user sessions, and access to HPC their own groups, and the single control it adds to the moderate baseline is AC-10, concurrent session control.

Patch inside thirty days

The control assumes a reboot is cheap and a package is a package. Neither holds. A node may be holding one rank of a job that waited days in the queue and has been running for days since; rebooting it does not patch a node, it deletes a week of someone's work. And the software on it is not a set of independent packages: the driver, the firmware and the fabric stack are certified together as a set, so moving one of them is a re-validation of all of them. What replaces the window is a mechanism. Drain nodes through the scheduler and patch them as they come free, roll the image rather than the package, and tie the disruptive work to allocation cycles the users already plan around. What you owe an assessor is the mechanism, its coverage, and the exception path for the thing that genuinely cannot wait.

Least privilege per user

The control assumes a permission is something a person holds. On a cluster the unit is the project. An allocation is granted to a project, its members share a directory on the parallel filesystem, and people join and leave while the allocation stays where it is. On top of that, research software frequently expects to build and run as root, because it was written somewhere that was free. What replaces it is project-scoped ACLs, so the grant follows the allocation and is revoked with it; enclave separation for regulated work, so one dataset's obligations do not become the whole machine's obligations; and no shared root anywhere, because an administrative account several people know is not an identity and cannot support any statement you later make about who did something. Role-based access control is the first of SP 800-234's fourteen groups, which is the same observation from the other side.

Back everything up

The control assumes storage is one thing. On a cluster it is at least three, and the fastest tier is deliberately not backed up. Scratch exists to absorb the write bandwidth of running jobs at petabyte scale: its economics do not permit a second copy, and its contents are meant to be regenerable from the inputs and the code that produced them. That is a control declined on purpose, not one overlooked, and writing "implemented" against it is worse than writing "no". What replaces it is a tiering policy that states out loud which data is recoverable and which is reproducible, and treats campaign storage differently from scratch. SP 800-223 describes campaign storage retention measured in years and archival retention measured in decades; those are different promises and they deserve different controls. Contingency planning for HPC is one of SP 800-234's fourteen groups for the same reason.

Change control and a software CAB

The control assumes changes are made by administrators, to systems you own, on a schedule a board approves. The most important software on a cluster is not changed that way. Researchers build their own stacks with Spack, Conda and EasyBuild, and bring containers built somewhere else entirely, on the cadence the science demands: a submission deadline, a collaborator's new version, a bug found that afternoon. A change advisory board placed in front of that either stops the work or, far more often, gets routed around and stops describing reality. What replaces it is control at the edges instead of in the middle: curated base images the site owns and rebuilds, a container policy built on Apptainer, which runs unprivileged and was designed for shared machines, and provenance on what actually ran, so the record answers the question the board was trying to ask. SP 800-234 gives user-developed software a group of its own.

The tailoring is no longer something each site invents

For most of the history of supercomputing, every site worked this out alone. That is why the standoff at the top of this post lasts six months: two people arguing from a baseline that was never written with this machine in mind, and no authority either of them can point at. Two NIST publications closed that gap, and they do different jobs.

NIST SP 800-223, High-Performance Computing Security: Architecture, Threat Analysis, and Security Posture, was finalized on 9 February 2024. It gives the machine a shared vocabulary: four function zones (the high-performance computing zone, the data storage zone, the access zone and the management zone), a threat analysis organized zone by zone, and a set of posture recommendations, one of which is explicitly about achieving security while maintaining HPC performance. It carries no control requirements, describing itself as a conceptual guide rather than a checklist of requirements. It also never mentions SP 800-53, worth knowing before anyone cites it in an argument about controls.

NIST SP 800-234, the High-Performance Computing (HPC) Security Overlay, was finalized on 4 May 2026, and it is the document that settles the argument. It takes the moderate baseline from SP 800-53B as its base and tailors on top of it.

The one control not carried over is AC-10, concurrent session control, which the overlay adds and tailors at the same time: that is why the denominator is 288 rather than 287. The tailoring is applied at five levels: the whole system, and then each of the four zones separately, so a control can mean one thing at the access zone and another inside the computing zone. The fourteen group titles are themselves a map of where the friction is: HPC logging, user sessions, contingency planning, network connections, user-developed software, shared GPUs and accelerators, the impact of controls on HPC performance and scalability, and a group titled inapplicable to HPC.

Both publications are developed under NIST's FISMA authority, which binds federal information systems, and both state that they may be used by nongovernmental organizations on a voluntary basis. So whether the cluster sits in a university, a research hospital, a pharmaceutical company or an AI team inside an ordinary business, this is available as a defensible answer rather than a local opinion, which is the part the standoff was missing.

We have written up both documents in full: NIST SP 800-223 for the reference architecture and the threat analysis, and NIST SP 800-234 for the overlay and how its sixty tailored controls are organized. If that standoff is sitting in your queue, the translation is the work our HPC security practice does: the assessment on the machine itself, the zoned architecture underneath it, and every control turned into something an assessor accepts and the machine survives.