Most high-performance computing (HPC) centers have built their scheduling culture around a particular kind of job: tightly coupled MPI simulations with fairly predictable runtimes. These jobs are scheduled across many nodes and are often checkpointed periodically. The checkpoints mean that if something fails, you can restart the job without losing the solution up to that point. In other words, a hardware failure might cost you hours instead of days. Resource managers such as Slurm, PBS, Grid Engine, and LSF have been used for this kind of workload for a long time.

This approach is no longer viable. The same clusters that run climate models and computational chemistry are also being asked to run artificial intelligence and machine language (AI/ML) training, notebooks, and inference workloads. Some of these workloads look a lot like traditional HPC batch jobs. Others have very different requirements around interactivity, elasticity, or latency. The result is not just a technical mismatch but perhaps a bit of a culture clash between HPC and AI communities that have learned to think about resources in different ways.

In this article I want to look at why mixed workload scheduling is difficult, where traditional schedulers can struggle, and some of the approaches being used to solve these problems. One of the more interesting solutions is Slinky [1], which is still a relatively new tool.

Before getting into the details, I want to make one thing clear: AI is not a single workload type. Training, interactive development, and inference can all behave differently. These distinctions matter when you start talking about scheduling. Additionally, HPC is not a single workload type. In addition to batch jobs, interactive HPC jobs are increasingly becoming an important component (HPC Desktop [2]).

Two Very Different Job Personalities

Ahead of scheduler mechanics, it helps to look at the workloads themselves. Table 1 gives a rough comparison. As always, you can find exceptions, but I think the differences are useful when thinking about scheduling policy.

Table 1: HPC Workloads

The deeper issue is not just that these jobs look different on paper, but the people submitting them can also have very different ideas about what “fair” scheduling means. A computational fluid dynamics user might expect a 256-rank job to start as a unit or not at all. Someone running a machine learning hyperparameter sweep might expect to launch dozens of smaller jobs and have them use whatever resources become available, but all of the jobs do not have to start at the same time. Both expectations make sense. The problem is that they don't always fit together very well.

Table 1 is a simplification. Distributed AI training can be just as fixed in size and coordinated as HPC. The important difference is that AI environments tend to include a wider range of workload types, not that every AI job is elastic.

Where Traditional Schedulers Strain

Slurm, PBS, Grid Engine, and LSF have traditionally been used with policies built around queued batch jobs, priorities, backfill, and fair share. These mechanisms do a good job of keeping a system busy while trying to balance competing users and projects. Once you put a wider range of AI workloads into the mix, however, some of the assumptions behind those policies start to show their age.

Long-Running Jobs that Camp on GPUs

A multiday, week, or month-long training job can occupy its GPUs continuously, which is not different from a long-running HPC job. They are both batch jobs. The interesting part is what happens when those jobs interact with the site's priority and fair share policies. If a few long-running training jobs consume a large share of a GPU partition for weeks, shorter jobs can have a harder time finding the idle gaps where they could have backfilled.

Preemption Is Harder than it Sounds

The obvious answer is to preempt the training job and let a higher priority job run. Configuring preemption is not too difficult and is supported by SchedMD’s Slurm [3]. Listing 1 is an example in a slurm.conf  file that has two partitions: The shorter jobs have higher priority and the longer running jobs have lower priority.

Listing 1: Two-Partition slurm.conf

# slurm.conf: enable partition-priority based preemption
...
# ====================================
# CONTROL AND PREEMPTION CONFIGURATION
# ====================================
SchedulerType=sched/backfill 
SelectType=select/cons_tres 
SelectTypeParameters=CR_Core_Memory
 
# Enable preemption based on Partition priorities
PreemptMode=REQUEUE
PreemptType=preempt/partition_prio
 
# ===========================
# PARTITION QUEUE DEFINITIONS
# ===========================
# 1. High-priority, short runtime queue
# This queue will kick off lower priority jobs if resources are full. PartitionName=short_high Nodes=node[01-10] Default=NO MaxTime=02:00:00 PriorityTier=100 PreemptMode=REQUEUE State=UP
 
# 2. Low-priority, long runtime queue
# Jobs here can run up to 7 days but will be requeued if short_high
# needs space.
PartitionName=long_low Nodes=node[01-10] Default=YES MaxTime=7-00:00:00 PriorityTier=10 PreemptMode=REQUEUE State=UP
...

A quick rundown of how preemption works [4] is that Slurm checks the priorities of the job and the quality-of-service (QoS) setting. If a higher priority job lacks free nodes, Slurm identifies lower priority jobs that are occupying those resources. Slurm has an optional grace period (GraceTime ) allowing low-priority jobs to save state before stopping.

Slurm provides for different preemption modes [5]:

• CANCEL  – terminates the lower priority job entirely
• REQUEUE  – stops the job and puts it back in the queue so it can run later
• SUSPEND  – pauses the job safely until resources are available
• GANG  – shares CPU and resources dynamically between jobs

In the configuration, the partition with a long runtime but lower priority is the default. The other partition, which is a much higher priority but shorter runtime, is not the default. The Slurm admin can control who has access to the higher priority queue.

This approach allows longer running jobs to be preempted so shorter running jobs can be run and works well if the preempted jobs can restart from a checkpoint. In this way, the previously used compute time is not wasted. If no checkpoints exist, the job cannot write or read checkpoints, or the job cannot be restarted, then preemption is pointless.

The good news is that checkpointing is much easier to add to many modern AI training workflows. Frameworks such as PyTorch Lightning and DeepSpeed provide capabilities that make it easier to save and restore training state. Preemption becomes more practical than it used to be. The catch, however, is that checkpointing is not free. Saving large training states can put a load on any shared storage and the I/O subsystem. Before turning on preemption, I would make sure the applications can restart correctly and that the storage system can handle the additional traffic.

Interactive Work Does Not Fit the Batch Model

Interactive work is another place where the traditional batch model starts to look awkward. Jupyter notebooks, inference endpoints, and interactive debugging are now common parts of an AI/ML environment. HPC users have also increased their use of interactive jobs, especially those using notebooks (see HPC Desktop [2]). None of these fits particularly well into a simple “submit the job and wait” workflow. If the user must keep checking the queue to see when an interactive session will start, the experience gets frustrating very quickly. (Who wants to stay up all night checking whether their interactive job started?)

Sites have responded to this dichotomy in a few different ways, including dedicated interactive partitions and QoS policies with appropriate limits. I don't think that necessarily means interactive work needs permanently dedicated hardware, but it does mean that notebooks, debugging sessions, and inference have requirements different from a long-running batch job, and the scheduling policies should recognize that point.

Strategies

Partitioning by Workload Type

The simplest mitigation, and probably the one most sites already use, is to split resources into different partitions or QoS tiers. That strategy lets you tune the scheduling policy for a particular workload instead of trying to make one policy work for everything. For example, in Slurm, you might create something like:

# Separate QoS definitions for simulation vs. training workloads
$ sacctmgr add qos sim_short MaxWall=04:00:00 Priority=100
$ sacctmgr add qos ai_long MaxWall=7-00:00:00 Priority=10 MaxTRESPerUser=gres/gpu=32

You can also tune the policies instead of letting the behavior emerge from a collection of scheduler settings. An allocation committee could, for example, decide that long-running training jobs should consume no more than a certain fraction of the available GPU hours during an accounting period.

Hybrid Schedulers and Slinky

As I have been discussing, the HPC world, because it is dominated by batch jobs, uses SchedMD’s Slurm. The AI world, where, in addition to training jobs that are really batch jobs with lots of shorter running jobs or interactive jobs, uses Kubernetes.

One of the more interesting developments in mixed workload scheduling is SchedMD's effort to bridge Slurm and Kubernetes directly [6]. The project is called Slinky, and the 1.0 release arrived in late 2025 after an early-access and release candidate period, with subsequent releases since then. SchedMD is now part of NVIDIA, having been acquired in December 2025. I think Slinky is worth watching because it takes a different approach to the problem: Instead of maintaining separate Slurm and Kubernetes pools, it tries to let them both work with the same underlying infrastructure.

Slinky has two main pieces: slurm-operator  [7] and slurm-bridge . The slurm-operator  Kubernetes operator runs Slurm inside a Kubernetes cluster. It manages Slurm controller and compute node pods as Kubernetes resources and can autoscale Slurm compute nodes in response to queue load. The slurm-bridge  component goes in the other direction. It allows Slurm to schedule Kubernetes pods as well as traditional Slurm jobs, so the same physical resources can potentially be used by both kinds of workloads.

Historically, a site that wanted Kubernetes for its AI/ML environment and Slurm for traditional HPC had a fairly simple choice: Pick one and accept its limitations or build separate pools of hardware. Slinky is an attempt to remove that hardware boundary. Although useful, it is important to understand what it does not do. Sharing the same physical resources does not magically answer the question of how the workloads should compete for those resources. You still need policy.

Slinky is still developing. The 0.x releases were useful for proving the architecture, but they also exposed some of the rough edges you would expect in a new project. The 1.0 release made important improvements around the API, upgrades, and disruption handling. Those steps are good for production use, but the operational experience at large HPC sites is still much newer than the experience with conventional Slurm deployments.

For an HPC site already invested in Kubernetes for its AI/ML environment, Slinky is interesting enough to pilot. For a site with a mature, heavily customized Slurm environment and little or no Kubernetes investment, I would probably watch Slinky for a while before making it the foundation of the production environment. The technology is moving quickly, and the operational playbooks are still being developed.

Other Convergence Efforts

Slinky is not the only project trying to close the gap. Volcano [8], a Cloud Native Computing Foundation (CNCF) project, brings batch scheduling, queueing, and gang scheduling concepts to Kubernetes without involving Slurm.

Slurm itself also has support for heterogeneous jobs, which can be useful when a job has different resource requirements in different stages. For example, a workflow might have a CPU-heavy preprocessing stage followed by GPU-heavy training. None of these approaches removes the need for site-specific policy, but they do show that mixed workloads are becoming a first-class scheduling problem rather than an edge case.

The Fair Share and Accounting Problem

Even if you get the scheduler mechanics right, you still encounter an accounting question. How do you charge GPU hours when one workload runs in short, predictable bursts and another one runs continuously for days or weeks? A simulation group that submits 10 four-hour jobs in a week looks very different in a fair share ledger from an ML group running one job continuously for the same total number of GPU-hours. The resource consumption might be almost identical, but the scheduling behavior is very different.

Most sites will end up dealing with this through policy rather than scheduler configuration alone. You might have separate allocation pools, periodic reviews by an allocation committee, or GPU-hour limits that are independent of job count and wall time. The scheduler can enforce pieces of these rules, but it cannot decide what the institution thinks is needed or fair. That part still requires people to sit down and agree on the policy.

Practical Recommendations

As with software stack fragmentation, no single fix exists. A few practices can make the situation easier, though, and usually without too much name calling (Table 2).

Table 2: Mitigation Strategies

If I were starting from scratch, I would consider the following:

• Start with partitioning, not preemption. Splitting GPU capacity by workload type is relatively low risk and reversible. I would add preemption only after confirming that the workloads can checkpoint and resume safely, which is not the most efficient in terms of resources, but it is easy.
• Pilot Slinky on a small node pool before committing to it. Version 1.0 is an important milestone, but it is still a new project. Treat it as an evaluation rather than as a drop-in replacement for a mature Slurm deployment.
• Bring the allocation committee into the scheduling conversation early. Changes to fair share and GPU-hour accounting can affect a researcher's perception of fairness just as much as changes to the scheduler itself.

Summary

Unlike software stack fragmentation, in which tools such as Spack, Lmod, and Apptainer give administrators a fairly clear path forward, mixed workload scheduling is still an open problem. The tools are improving, but the policies are often trying to catch up. Partitioning by workload type is a reasonable place to start. Preemption and checkpoint-aware scheduling can close more of the gap, but only when the applications are built to handle it. Projects such as Slinky offer another interesting approach by letting Slurm and Kubernetes share a converged cluster rather than forcing a choice between them. Slinky is still young enough that “watch closely” is probably better advice for some sites than “adopt now.”

If come away with one practical point, it is this: Don’t expect one scheduler setting to solve the tension between batch HPC and AI workloads. The technical mismatch is real but so is the difference in how the users think about resources. The sites that make the most progress will probably be those that treat this problem from both an engineering and a policy perspective.

References

[1] Slinky: Slurm workload management for Kubernetes: https://www.nvidia.com/en-us/software/slinky/

[2] “HPC Desktop” by Jeff Layton. ADMIN , June 2026: https://www.admin-it.io/hpc-desktop/

[3] SchedMD, Slinky documentation: https://slinky.schedmd.com/docs/

[4] SchedMD. Slurm Workload Manager documentation: heterogeneous jobs and preemption: https://slurm.schedmd.com/

[5] SchedMD preemption: https://slurm.schedmd.com/preempt.html

[6] Arnold, Nathan. “Running Slurm on Amazon EKS with Slinky.” by Nathan Arnold. AWS Containers , October 2025: https://aws.amazon.com/blogs/containers/running-slurm-on-amazon-eks-with-slinky/

[7] Kubernetes operator for Slurm clusters: https://github.com/SlinkyProject/slurm-operator

[8] Volcano: https://volcano.sh/