Vendor-neutral orchestration for AI accelerators · technical thesis
Alloy routes every AI workload to the cheapest silicon that meets its SLO — with hard, audited isolation — across NVIDIA, AMD Instinct, and AWS Trainium in one Kubernetes-native layer. This page lays out the thesis, the research it stands on, and early simulation results.
one device · eight isolated tenants
MI300X in CPX mode — 8× inference density per card
AI hardware stopped being a one-vendor story: high-memory AMD parts, custom cloud silicon, and CPU tiers now sit beside NVIDIA in real production fleets — driven by price, memory capacity, availability, and board-level supplier-risk mandates. But the scheduling layer never caught up. Each accelerator family arrives with its own tooling stack, its own queue, its own stranded capacity.
A vendor-owned scheduler cannot fix this, structurally: it will never route honestly onto a competitor's hardware. Neutrality isn't a feature a vendor can add — it's a property of who owns the layer. Alloy's position: accelerator choice should be a per-workload routing decision, made by an independent layer, explainable on demand.
Different chips genuinely differ — partition granularity, memory bandwidth, interconnect topology. Pretending they're interchangeable wastes the best silicon; pinning workloads per vendor strands the rest.
Per-vendor pools mean per-vendor queues: one pool saturates while the next idles at zero. Capacity is stranded not by demand but by the inability to route across the seam.
Fractional sharing without enforcement is a noisy-neighbor incident waiting to happen. Hard, audited isolation on non-CUDA silicon is the unsolved, defensible engineering problem.
Accelerators bill by the hour whether they work or not. Across production Kubernetes clusters, the median GPU sits idle the vast majority of the time — not for lack of demand, but because scheduling is naive: whole cards allocated to fractional jobs, vendor-pinned queues, gangs blocking capacity they can't yet use. Alloy monetizes the delta between the silicon you paid for and the silicon you actually use — across all vendors simultaneously.
Drop-in schedulerName: alloy. Gang scheduling with backfill, per-vendor topology models, fair-share quota trees with borrowing. Built on upstream DRA and the scheduler-plugin framework — never a fork.
Every fraction Alloy hands out is backed by a hardware or driver-level enforcement primitive — MIG, CU masking, CPX partition modes, NeuronCore pinning — continuously audited, honestly tiered as hard / enforced / soft. Isolation you can show your auditor.
Accelerator type is a routing decision, not a constraint. Fingerprint the workload, predict throughput per silicon, place on the cheapest option that meets the SLO, watch quality gates, and record every incompatibility in a compounding knowledge base.
Hierarchical quotas, chargeback and showback, RBAC/SSO, policy-as-code, multi-cluster federation, air-gapped builds for sovereign deployments.
Every placement emits this decision record — candidates, scores, gates. Exportable to a model-risk archive. No vendor pays for placement priority. Neutrality is a governance artifact, not a slogan.
We built a trace-driven, discrete-event simulator and replayed a production-shaped workload — 4,000 jobs whose arrival patterns, size skew, and heavy-tailed durations follow the published Microsoft Philly trace characteristics — onto an identical 112-device mixed fleet (H100, A100, MI300X in CPX mode, Trainium). Two policies compete: a baseline that models the absence of orchestration (FIFO, whole devices, every job on its fastest vendor-pinned pool) and Alloy's cost-routed policy (effective-cost routing, fractional packing, backfill).
The baseline's pathology is the familiar one: everything piles onto the premium pool (86% busy, a 17-day queue behind it) while three other pools sit idle — capacity stranded not by demand but by the inability to route across the vendor seam. The routed policy engages every pool by effective cost and dissolves the queue.
The academic record solved heterogeneity-aware routing and single-vendor sharing as separate problems. No published system does fractional allocation with enforcement across vendor runtimes under one placement brain. That composition is Alloy. The briefs below are our working notes on the load-bearing papers — every one is open access.
Different models speed up differently on different silicon — ResNet-50 gains ~10× moving K80→V100 while a deep-RL model gains 2×. Gavel measures a throughput matrix per (job × accelerator), rewrites classic policies as optimization problems over it, and improves completion time up to 3.5×. It's the academic blueprint for Alloy's placement engine — we extend the matrix to workload fingerprints, fractional devices, and cross-vendor enforcement.
MAPS TO → placement engine · fair-share policy math
MPS and MIG demand manual limits and fixed profiles. TGS shares GPUs beneath the container with adaptive rate control — throttling opportunistic jobs until the production job runs at solo speed, no app changes. Production performance within a few percent of exclusive; up to 7.8× more opportunistic throughput than MPS. The correct architecture for Alloy's soft tier, and its feedback loop doubles as our conformance monitor.
MAPS TO → isolation runtime (soft tier) · enforcement auditing
The first honest public look inside a production GPU cluster: gang jobs queue for simultaneous availability, allocated GPUs run far below capacity, ~30% of jobs fail — with failures clustering on specific unhealthy nodes that burn GPU-hours on retries. The empirical foundation for backfill, allocated-vs-used showback, and flaky-node quarantine. The trace is public; our simulation methodology follows its published workload shape.
MAPS TO → economic thesis · gang + backfill · health-driven scheduling · §04 methodology
Quota counted in raw GPUs breaks gang jobs: a tenant "owning" 8 GPUs can receive 8 useless fragments — the sharing anomaly. HiveD denominates quota in topology-shaped cells managed by buddy allocation, so borrowing preserves affinity. Alloy extends cells one level down — into MIG slices, XCDs, and NeuronCore groups — making the fractional sharing anomaly structurally unrepresentable.
MAPS TO → quota trees · topology tiers · fractional cells design
Even the industry's gold-standard hard partition leaks: shared TLBs (31 kb/s covert channel, CCS '23), shared PCIe paths that fingerprint a neighbor's LLM with 93% accuracy (MICRO '24), and now L2-cache side channels on H100-class MIG. This is why Alloy tiers isolation honestly — hard / enforced / soft, documented against known attack classes — and audits every partition continuously instead of trusting the datasheet.
MAPS TO → tenancy tiers · conformance auditing · multi-tenant contracts
Once GPUs are shared fractionally, classic bin-packing lies: a node looks full while its leftover slivers fit no real workload. Alibaba defines fragmentation relative to the live workload mix and schedules by fragmentation gradient descent — recovering double-digit percentages of stranded capacity in production. FGD is Alloy's default packing score, extended across vendors.
MAPS TO → binpack plugin · fragmentation index dashboard
Also in the corpus: Sia (SOSP '23) on online-learned throughput models, Pollux (OSDI '21) on goodput, Gandiva, Tiresias, AntMan, DRF, Guardian, and the dynamic-MIG literature.