Supercharge Your LLM performance with Amazon Sagemaker Model Model Model Defer V15

Today, we are very happy to announce the Amazon Sagemaker Model Model Model Model Model (LMI) Buy v15, enabled by VLLM 0.8.4 for Vll V1 engine support. This translation now supports the latest source models, such as Meta's Llama Models 4 and Maverick, Gemma 3, QWEN's QWEN, Deepseek-R, and much. Amazon Sagemaker AI continues to wake up its AI's property skills.

This release introduces important performance development, advanced compliance with multimodification (ie, the ability to understand and evaluate the text, and to-photos (Text-to-photos (photos (Text-to–to-photos (photos (Text-to–to-photos (photos (Text-to–to-photos (photos (Text-to-photos (photos (Text-to–to-photos (photos (Text-to–to-photos (photos (Text-to–to-photos (photos (Text-to–to-photos in the most significant performance.

What's new?

LMI V15 brings several enhancements that improve the issuance, latency, and useful:

1. Async mode that is directly associated with asyncLllmengine of Asynmclmmngine for improved management. This mode creates a very effective post-continuous process, enables manufacturing multiple applications and outgoing distribution by high anactions in Acoking than previous BATCH.
2. VLMM V1 engine support, bringing higher 111% filling with comparison to the previous v0 engine for small models with high encounter. This development development appears in reducing the CPU overhead, designed for resources, and the use of more resources in the construction of the V1. LMI V15 supports both V1 and V0 engines, with V1 being automatic. If you have a need to use V0, you can use the V0 engine by specifying VLLM_USE_V1=0. VLLM V1 engine has come with the sophistication of the most efficient engine in simplified engine programming, associated zero-overhepsy maintenance, advanced preparation and more information, see additional information, see Vlll Blog.
3. Developed API Schema support for three variables for allowing fraudulent integration on applications built in famous API patterns:
   1. Message format is compatible with Opelai Chat Compions API.
   2. An openai's completion format.
   3. The Textraction Infence (tgema) Support the right hand compatibility and adult models.
4. Multimodal Support, Developed Support Language model of vision including efficiency as multimodal Prefix Caching
5. The built-in support for the calling and driving tool, enables the performance of agent-based function.

Advanced model support

LMI V15 supports an expandable system of land-of-the-the-art, including recent releases from the lead model providers. The container provides a fixed compatibility to be compatible but not limited to:

• 4th 4 – Llama-4-Scouts-17B-16e and LLAMA-4-Maverick-17b-128E
• Gemma 3 – Light and simple models are very simple, unknown about their solid performance despite small size
• QWEN 2.5 – The developed models of Alaba puts the QWQ 2.5 and a lot of powerful QWEN2-VL
• Models Ai Ai Ai Ai Ai – Well-efficient models from the mysterious AI give effective measurement and special skills
• Deepseek-R1 / V3 – Status of Models of Art of Art

Each model family can be sent using the LMI V15 container by specifying the appropriate model ID, for example, meta-llama / icocial-16b-16e, and is prepared for environmental parameters or planning.

Benches

Our benchmarks show the working business benefits of the engine of the LMI V15's V1 compared to previous versions:

Deepseek-R1 Llama 70B of various conceurriency levels

LLAMA 3.1 8B commands to a variety of concurrency degree

MISTRAL 7B of various Levels of ConnlyRency

The AsyNC Elim V15 engine shows power in high conditions, where many applications at the same time benefits from handling the prepared application. These benches highlight that the V1 engine in Async's mode moves between 24% and 111% prepnection of comparison with LMI V14 uses continuing to remember the following performance.

• The upper batch sizes increases the consent of permission but also a natural trading in terms of latency
• The 4 and 8 batch sizes provide the best latency to use most cases
• Batch size up to 64 and 128 to achieve higher fillings with acceptable sales of latency

API formats

LMI V15 supports three API schemas: Elimination of Openi conversation, TRIGHT CLICK, AND THE TGI.

• Discussion for discussion – Message format is compatible with Opelai Chat Compions API. Use this schema for driving tool, consultation, and multimaloral charges. Here is the bustper sample with Messages API:

  body = {
      "messages": [
          {"role": "user", "content": "Name popular places to visit in London?"}
      ],
      "temperature": 0.9,
      "max_tokens": 256,
      "stream": True,
  }

• Opening Division Format – The Compliment API ENDPONT is no recipient:

  body = {
   "prompt": "Name popular places to visit in London?",
   "temperature": 0.9,
   "max_tokens": 256,
   "stream": True,
  } 

• Tgi – Supports backward confinement in old models:

  body = {
  "inputs": "Name popular places to visit in London?",
  "parameters": {
  "max_new_tokens": 256,
  "temperature": 0.9,
  },
  "stream": True,
  }

Starting with LMI V15

Getting started with LMI V15 No seams, and you can move LMI V15 with a few lines of the code. The container is available through the Amazon Elastic Dinater Registry (Amazon ECR), and submission can be managed through Sagemakers Ai EDPoints. Sending models, you need to specify the facial operation ID, the type of configuration and configuration as environmental variety.

By working well, we recommend the following situations:

• Llama 4 Scout: ML.P.48XGage
• Deepseek R1 / V3: ml.0.48xigerge
• QWEN 2.5 VL-32B: ML.G5.12XGERGE
• QWEN qwq 32B: Ml.g5.12xigerge
• Bad evil: ml.g6e.48xgage
• Gemma3-27b: Ml.g5.12xcharge
• LLAMA 3.3-70B: ML.P4D.24XGERGE

To move with LMI V15, Follow these steps:

1. Clone Notebook in Amazon Sagemaker Studio Notebook or Visual Studio code (VS code). You can start the writing book to make the first setup and send the model from Hugging Faces to Saghemaker Ai Endpoint. We walk in key blocks here.
2. LMI V15 keeps the same configuration pattern with previous versions, using natural flexibility OPTION_. This consistent approach makes it directly to users familiar with previous LMI versions in the migration in v15.

   vllm_config = {
       "HF_MODEL_ID": "meta-llama/Llama-4-Scout-17B-16E",
       "HF_TOKEN": "entertoken",
       "OPTION_MAX_MODEL_LEN": "250000",
       "OPTION_MAX_ROLLING_BATCH_SIZE": "8",
       "OPTION_MODEL_LOADING_TIMEOUT": "1500",
       "SERVING_FAIL_FAST": "true",
       "OPTION_ROLLING_BATCH": "disable",
       "OPTION_ASYNC_MODE": "true",
       "OPTION_ENTRYPOINT": "djl_python.lmi_vllm.vllm_async_service"
   }

   • HF_MODEL_ID It puts the model ID from the face of face. You can also download the model from Amazon Simple Service (Amazon S3).
   • HF_TOKEN Puts the token to download the model. This is required in the Gated Models Like Llama-4
   • OPTION_MAX_MODEL_LEN. This is a model model of Max model.
   • OPTION_MAX_ROLLING_BATCH_SIZE Puts batch size with model.
   • OPTION_MODEL_LOADING_TIMEOUT It puts the amount of the Sagemaker output time to load the model and run health checks.
   • SERVING_FAIL_FAST=true. We recommend setup this flag because it allows the Sagemaker to rebuild the bush where the default engine occurs.
   • OPTION_ROLLING_BATCH= disable It disables the batch use of LMI LMI, which was default in LMI V14. We recommend using Async instead of recent launch and provides better performance
   • OPTION_ASYNC_MODE=true Enables async mode.
   • OPTION_ENTRYPOINT provides for the covering of vllm's Async integration
3. Set the latest bowl (this exemplary 0.33.0-lmi15.0.0-cu128), The Region of AWS (us-east-1), then create model art with all configuration. Reviewing the latest version of the container, look at the deepest images of deep reading.
4. Use model in a perfect place using model.deploy().

   CONTAINER_VERSION = '0.33.0-lmi15.0.0-cu128'
   REGION = 'us-east-1'
   # Construct container URI
   container_uri = f'763104351884.dkr.ecr.{REGION}.amazonaws.com/djl-inference:{CONTAINER_VERSION}'
   
   # Select instance type
   instance_type = "ml.p5.48xlarge"
   
   model = Model(image_uri=container_uri,
                 role=role,
                 env=vllm_config)
   endpoint_name = sagemaker.utils.name_from_base("Llama-4")
   
   print(endpoint_name)
   model.deploy(
       initial_instance_count=1,
       instance_type=instance_type,
       endpoint_name=endpoint_name,
       container_startup_health_check_timeout = 1800
   )

5. Ask Model, Sagemaker Infence provides two apis to request a model- InvokeEndpoint including InvokeEndpointWithResponseStream. You can select an option based on your needs.

   # Create SageMaker Runtime client
   smr_client = boto3.client('sagemaker-runtime')
   ##Add your endpoint here 
   endpoint_name=""
   
   # Invoke with messages format
   body = {
   "messages": [
   {"role": "user", "content": "Name popular places to visit in London?"}
   ],
   "temperature": 0.9,
   "max_tokens": 256,
   "stream": True,
   }
   
   # Invoke with endpoint streaming
   resp = smr_client.invoke_endpoint_with_response_stream(
   EndpointName=endpoint_name,
   Body=json.dumps(body),
   ContentType="application/json",
   )

Implementation of many views with lllama-4 Scout, see a sample code of the full code of employment requests.

Store

Amazon Sagemaker LMI consisting of V15 represents an important step forward to the huge skills of the nominations. With the new VLM V1, Async operating mode, Async mode, expected support, efficient performance, can send the determination of llms for a major functioning and flexibility. Configurable options of a container gives you a flexibility of good shipping of certain needs, even if they do the latency well, the full, or cost.

We encourage you to check this release by sending your generating AI models.

See the written textbells provided to start the lmi v15.

About the authors

Vivek Gazasani Is the construction of special acquisitions of AWS. It helps the AI ​​emerging companies create new solutions to the AWS services and the Durben Compute. Currently, focus on the development of good planning and efficiency in the operation of large languages. In his free time, Vivek enjoys walking, watching movies, and trying different cuisines.

Siddharth Venkatesan Software engineer to deep AWS reading. You are currently focused on the building solutions of the great model. Before AWS worked in Amazon Grocery Org to create new features of payment to customers around the world. Without work, he enjoys skiing, outdoors, and looking at sports.

Felipe Lopez You are a special AI / ML Specialist Solutions by AWS. Before joining the AWS, Felipe worked with a digital ge digital and SLB, where they focus on modeling products and the efficiency of industrial applications.

Baby Nagasundaram It leads the product, engineering, and relationships with Amazon Sagemaker JumpStart strategies, Sagemaker Machine Learning and Ai Hub. You are passionate about building solutions that help customers accelerate their AI and open the business value.

Dmitry designed Is the construction of the highest AI / ML solutions in Amazon Web Resources to services (AWS), to assist customers design and create AI / ML solutions. DMITRY's work includes a list of ML use, with primary interest in making a strong AI, learning, and measuring ml across the business. He has assisted companies in many industries, including insurance, financial services, services, and telecommunications. You can connect with DMITRY on LinkedIn.