The purchase order came out of an AI budget, against a training deadline that was set before anyone asked where the machine would live. It was racked, it was cabled, and it worked. The first runs finished. Some months later, a sentence reaches a security team that has heard nothing about any of it: we have just bought a GPU cluster and nobody owns its security.
That is a good sentence, because someone has finally noticed. It is also late: by the time it is said the machine has been in the building for a while, holding some of the company's most valuable data for most of it.
It arrived as an AI project
It arrived as an AI project rather than an infrastructure one, and that single fact explains nearly everything that follows.
Infrastructure gets bought through a process with security somewhere in it: an architecture review, a standard to build to, a person whose job includes saying no. A GPU cluster gets bought as capability, and the questions in that room were how many parameters and by when.
So it turns up with its own everything. Its own network, because a high-speed interconnect between the nodes is the entire reason it is a cluster rather than a pile of servers. Its own storage, because the GPUs have to be fed and the corporate file share cannot do it. Its own admins, the platform engineers who stood it up, who are usually very good and who do not report to security. And its own way in, which on a new cluster is frequently one login host and a key somebody generated the afternoon the racks were powered on. In the register it shows up as a hardware purchase, or as a cloud account with a spend alert on it, or as nothing. What it does not show up as is a system: a thing with zones, users, data, a boundary and a named owner.
The ownership vacuum that follows is quiet and symmetrical. The AI team assumes the machine inherited whatever protects the rest of the estate, because everything else they use did. The security team, if it knows the cluster exists, assumes the standard controls landed on it the way they land everywhere. Both assumptions are checkable in an afternoon, and both are usually wrong.
None of this is negligence. It is what happens when a machine that behaves like a supercomputer is procured like a project.
What is actually on it
Sort HPC systems by the data they carry, and most of the categories come with a regime attached. Universities and research hospitals hold human-subjects and genomic data under ethics approvals and controlled-access agreements. Health systems carry patient data under the privacy regime it was collected under. National labs and defense suppliers hold export-controlled research under ITAR and EAR.
Then there is the category this post is about: models and training data, held by AI teams inside ordinary companies. What binds it is no statute at all. A customer contract, the training data, and the worst day the company has if it leaks.
Three things live on the machine, and they are worth separating because they fail differently.
The training data is the one nobody registered. It is rarely born on the cluster; it is assembled, from customer records exported from production, support transcripts, logs, documents a partner shared under an agreement, a dataset licensed with terms attached. Every one of those had conditions where it came from, and the conditions travel with the copy. Controlled material has been reassembled, in bulk, on a system that was in scope for none of the reviews the originals passed.
The model weights are the accumulated output of every GPU-hour the company has bought. If the machine holds anything that behaves like an asset on a balance sheet, it is this, and it is a file.
Around both sits the working state: checkpoints, staged copies of the input, intermediate output on scratch. This is the layer that undoes a clean architecture in practice, the copy nobody remembers making, on the tier that is fast precisely because it is not asking many questions.
The absence of a statute reads like relief and is closer to the opposite. A regulated dataset comes with an assessor, a clause number and a date, which is unpleasant and also organizing: it tells you when the work has to be finished. Here nothing external sets that deadline, so the obligation is entirely real and entirely internal, which is the shape of work that stays on a plan.
Why the enterprise stack misses it
Your controls assume a host you administer, software you approved, and a machine you can reboot. A cluster gives you none of the three, and a GPU cluster gives them up for the same structural reasons a national lab's machine does.
The EDR agent on every host. The control assumes a host with spare capacity doing work that belongs to it. A node in a distributed training run is neither: the run is barrier-synchronized, each step ends in a collective where every rank waits for the slowest, and an agent that wakes on its own cadence on one node delays the step on all of them. That effect is measured rather than theoretical, documented at scale in Petrini, Kerbyson and Pakin's "The Case of the Missing Supercomputer Performance" (SC 2003). License and telemetry then multiply by node count. What replaces it is eBPF and sampled instrumentation, out-of-band collection, and telemetry drawn from the fabric and the scheduler, which already know which account ran what and where.
Patch inside thirty days. The control assumes a reboot is cheap and a package is a package. On this machine a node may be holding one rank of a run that has been going for days, so the reboot is not a maintenance action, it is a decision to lose that work. And the software is not a set of independent packages: the driver, the firmware and the fabric stack are certified together as a set, and on a GPU cluster that set extends to the container toolkit and the collective communications library the training code is built against. Moving one is a revalidation of all of them. What replaces the window is a mechanism: rolling drain-and-patch through the scheduler, image-based reprovisioning, and disruptive maintenance tied to cycles the users already plan around. What you owe an assessor is that mechanism, its coverage and its exception path, not a number the machine was never going to meet.
Change control and a software CAB. The control assumes changes are made by administrators, to systems you own, on a schedule a board approves. The software that matters here is not changed that way: environments are built on the machine, containers arrive from public registries, weights are pulled from a hub, and the cadence is whatever the run needs this week. A board placed in front of that either stops the work or, far more often, gets routed around and stops describing reality. What replaces it is control at the edges rather than in the middle: curated base images the platform team owns and rebuilds, an explicit container policy, and provenance on what actually ran. SP 800-223 treats containerized software as part of the architecture rather than an exception to it, naming Docker, Apptainer and Charliecloud among the container runtimes and Kubernetes among the orchestration tools.
The zone architecture, applied to one GPU cluster
There is a published architecture for this class of machine, and it is worth being precise about what it is and is not. NIST SP 800-223, High-Performance Computing Security: Architecture, Threat Analysis, and Security Posture, was finalized on 9 February 2024. It divides an HPC system into four function zones, adds a threat analysis organized zone by zone, and closes with posture recommendations, one of which is explicitly about achieving security while maintaining HPC performance. It describes itself as a conceptual guide rather than a checklist of requirements. It was developed under NIST's FISMA authority, which binds federal information systems, and it states that it may be used by nongovernmental organizations on a voluntary basis. Nothing in it requires anything of a company that bought a GPU cluster.
What it offers instead is a shape, and the shape fits, because the machine is the same shape: compute nodes on a high-speed interconnect, a shared parallel filesystem, a scheduler, a login path, and a management plane that runs the whole thing.
Outside the cluster
The corporate network and the identity provider. This is the part your existing stack already covers, and the part most people assume the cluster is inside.
Login and jump hosts, the notebook or scheduler front end, and the path training data arrives on and results leave by. On a new cluster this is usually one box and a static key, reached over SSH.
The GPU nodes and the interconnect between them. Provisioned from one image, handed back to the scheduler at the end of a run, with accelerator memory that holds state until something clears it.
The parallel filesystem the GPUs are fed from, plus whatever is kept longer. Training data, checkpoints and weights, on tiers with different promises attached to them.
Provisioning, configuration management, the out-of-band network and the scheduler's own control plane. SP 800-223 describes this zone as carrying a separate security posture, reached through a bastion host or VPN on networks not routed beyond the perimeter, with configuration management more restricted still.
The value of drawing it is that a question nobody could answer becomes four questions somebody can. Who may arrive, and how, is an access zone question. What the nodes retain between tenants is a computing zone question. Where the training data rests, and which copies are legitimate, is a data storage zone question. Who can reprovision the whole machine, and from where, is a management zone question, and on a cluster stood up quickly it is the one most likely to have a temporary answer.
The control layer came later. NIST SP 800-234, the High-Performance Computing (HPC) Security Overlay, was finalized on 4 May 2026. It takes the moderate baseline from SP 800-53B as its base: 288 controls appear in its table, 287 carried from that baseline plus AC-10, concurrent session control, which the overlay adds. Sixty are tailored with supplemental guidance, discussion or both, sorted into fourteen thematic groups. The detail that matters here is where the tailoring is applied: at five levels, the whole system and then each of the four zones separately, so one control can mean one thing at the access zone and something else inside the computing zone. One of the fourteen groups is shared GPUs and accelerators, which addresses this machine's defining component by name. The overlay carries the same voluntary-adoption language, and its introduction expects customization rather than verbatim adoption.
The first three moves
The goal is to bring the cluster into the estate on purpose rather than by accident. Three moves get you most of the way, in this order.
Inventory and ownership. Register the cluster as a system rather than as hardware: its four zones, the data classes on it, who has accounts, what it connects to, and the boundary you are claiming. Then name two people, because one is never enough. Someone owns how controls are implemented, which requires the ability to change the images, the scheduler configuration and the filesystem, so they sit with the team that runs the machine. Someone owns the standard it is held to and the record of what it does. A machine with one nominated owner who controls half of it does not have an owner, it has a volunteer.
Identity and segmentation designed for the cluster. Access is the control that survives contact best here, and the one most likely to still be in its day-one state. Replace static keys with short-lived credentials, enforce multi-factor at the access zone, which exists to authenticate and authorize arrivals, and scope permissions to the project rather than the person, because the project is the unit people join and leave. Then segment at the zone edge rather than in the hot path: RDMA lets one node's adapter write into another's memory without either kernel taking part, so an inline appliance on the interconnect is either bypassed or is reimposing the latency the fabric was bought to remove. And separate the management plane properly, off the general network and reached through a bastion, because the temporary arrangement from build week is usually still there.
Logging out of band. Per-host forwarding fails twice here: the volume multiplies by node count and swamps the pipeline and the license, and the record is the wrong shape even when it survives, because the accountable unit is the job rather than the host. A node runs one project's work and then another's, and a per-host line does not say which. Sample and aggregate at the fabric and the scheduler, ship it out of band, and key it to the job and the account. HPC logging is one of SP 800-234's fourteen groups for exactly this reason, which is a useful thing to point at when the request comes back as a request for an agent.
None of the three is exotic, and none is what the enterprise stack would have done on its own. The machine is not unsecurable. It is unaddressed.
Both publications are written up in full: NIST SP 800-223 for the reference architecture and threat analysis, NIST SP 800-234 for the overlay and its sixty tailored controls. If the argument in your queue is about individual controls, the companion post walks ten of them: why enterprise security controls break on HPC. If a dataset with conditions attached has landed on the machine, CUI on a shared HPC cluster covers the enclave pattern. And if the sentence at the top of this post has already been said out loud in your company, bringing the cluster into the estate on purpose is the work our HPC security practice does.