Amazon Web Services has added NVIDIA Multi-Instance GPU support to Amazon SageMaker Inference, allowing customers to split supported accelerators into smaller hardware-isolated partitions for model-serving workloads. The capability was announced on November 6 and can improve utilisation when a model does not need an entire GPU. AWS says customers can deploy multiple inference components on a single endpoint and assign each model a partition that matches its compute and memory requirements, instead of reserving a full accelerator for every model.
The problem AWS is addressing is common in production AI systems. A GPU endpoint is often provisioned in whole-device units, even when the model uses only part of the accelerator’s memory or processing capacity. A machine with several GPUs may So host fewer models than its hardware could theoretically support. Smaller models can leave capacity unused, while moving to a larger accelerator may provide more performance than a workload needs. Both situations can increase the cost of serving models.
NVIDIA Multi-Instance GPU, or MIG, divides a supported physical GPU into separate instances with dedicated portions of compute, memory and memory bandwidth. AWS says the partitions are isolated in hardware, which can help prevent one model’s workload from affecting another model’s performance. The feature is available for endpoints that use SageMaker inference components. AWS notes that single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned, so the capability does not apply to every SageMaker deployment pattern.
How GPU partitioning changes model hosting
AWS says customers configure a partition layout when creating a SageMaker endpoint and assign a partition profile to each inference component. Supported GPU families listed in the launch material include P4d with NVIDIA A100, P5 with H100, G7 with RTX PRO 4500 Blackwell and G7e with RTX PRO 6000 Blackwell Server Edition. The service adds optional fields to existing endpoint configuration calls rather than introducing a separate API or IAM action. Layouts are provisioned with the endpoint, so teams should choose the partition mix before deployment and validate it against the exact GPU profile.
The maximum number of partitions depends on the GPU and does not exceed seven, according to AWS. MIG layouts also have operational constraints. AWS says a layout can only be changed when partitions are idle, and on SageMaker the layout is selected when the endpoint is created. Models that require more than one GPU still need whole devices. These boundaries mean teams must measure model memory, concurrency and latency requirements before choosing a partition profile.
Workload fit and cost considerations
GPU partitioning is most useful when an organisation serves several models that each need less than a full accelerator. A portfolio of fine-tuned models, per-customer variants or smaller inference workloads may benefit from sharing hardware while retaining resource isolation. It may be less suitable for large models that need multiple GPUs or workloads with highly variable resource demands. The right configuration depends on the accelerator type, model size, request rate, prompt length and latency target.
More efficient utilisation does not automatically translate into a fixed percentage reduction in cloud spending. The financial result depends on how much unused capacity existed before partitioning, whether more models can be consolidated, and whether the new layout changes throughput or latency. Teams should benchmark realistic traffic, including peak concurrency and long prompts, rather than relying on average utilisation alone. They should also compare the cost of a partitioned newer accelerator with a whole older GPU.
AWS says Salesforce helped define requirements and validate the initial implementation. The feature is a practical addition to managed AI infrastructure because it exposes a hardware capability without requiring every customer to operate the full GPU partitioning stack. The next test will be how well it performs across supported instance families and regions. Customers should check regional availability, quotas, driver requirements and endpoint limitations before migrating production workloads. For suitable models, MIG can make inference capacity more granular and reduce the need to allocate a whole GPU to every endpoint.