Track and manage assets used in AI development with Amazon SageMaker AI

Building custom base models requires linking multiple assets across the development lifecycle such as data assets, computing infrastructure, model architecture and frameworks, genealogy, and production deployment. Data scientists build and refine training data sets, develop custom testers to test model quality and safety, and iterate through fine-tuning to improve performance. Since this workflow scales across groups and scenarios, tracking which specific dataset versions, tester settings, and hyperparameters generated each model becomes challenging. Teams often rely on manual entries in notebooks or spreadsheets, making it difficult to reproduce successful experiments or understand inventory models.

This challenge is intensified in enterprise environments with multiple AWS accounts for development, deployment, and production. As models pass through deployment lines, maintaining visibility into their training data, test criteria, and settings requires critical communication. Without automatic tracking, teams lose the ability to track used models back to their origins or share assets consistently across magic. Amazon SageMaker AI supports tracking and managing assets used in productive AI development. With Amazon SageMaker AI you can register and modify models, datasets, and custom testers, and automatically capture relationships and genealogies as you fine-tune, test, and deploy productive AI models. This reduces the subject of manual tracking and provides complete visibility into how models are created, from the base model through production deployment.

In this post, we'll explore new capabilities and key concepts that help organizations track and manage model development and deployment lifecycles. We'll show you how to configure features to train models with end-to-end automation, from dataset loading and transformation to optimization, testing, and endpoint deployment.

Managing data set versions across trials

As you refine the training data to create a custom model, you often create multiple versions of the data sets. You can register datasets and create new versions as your data evolves, each version being tracked independently. When you register a dataset in SageMaker AI, you provide an S3 location and metadata that describes the dataset. As you improve your data—or add more samples, improve quality, or adjust for specific use cases—you can create new versions of the same dataset. Each version, as shown in the following image, stores its own metadata and S3 location so you can track the evolution of your training data over time.

When you use a dataset for optimization, Amazon SageMaker AI automatically connects a specific version of the dataset to the resulting model. This supports comparisons between models trained on different versions of the data set and helps you understand which improvements to the data led to better performance. You can also use the same dataset version across multiple tests to gain consistency when testing different hyperparameters or optimization techniques.

Creates reusable custom testers

Testing custom models often requires domain-specific quality, security, or performance criteria. A custom tester contains Lambda function code that receives input data and returns test results including scores and validation status. You can define testers for a variety of purposes—checking response quality, testing safety and toxicity, verifying output format, or measuring the accuracy of a specific task. You can track custom testers using AWS Lambda functions that run your test logic, and then convert and use these testers again across models and datasets, as shown in the following image.

Automated line tracking throughout the development lifecycle

SageMaker AI dynamic range tracking automatically captures relationships between assets as you build and test models. When you create a fine-tuning job, Amazon SageMaker AI associates the training job with input datasets, baseline models, and output models. When performing testing activities, it includes analysis on the models being tested and the testers being used. This automatic list capture means you don't need to manually write which properties are used in each test. You can view the complete list of the model, which shows its base model, training data sets with specific versions, parameters, test results, and deployment locations, as shown in the image below.

With list view, you can trace any used models back to their origin. For example, if you need to understand why a production model behaves in a certain way, you can see exactly what training data, optimal configuration settings, and test criteria were used. This is especially important for administrative, reproducibility, and debugging purposes. You can also use the list information to reproduce the tests. By identifying the exact dataset version, tester version, and configuration used for a successful model, you can recreate the training process with confidence that you are using the same input.

Integration with MLflow for test tracking

Customization capabilities for Amazon SageMaker AI models with automated behavior integrated with SageMaker AI MLflow Apps, which provide automated coordination between model training tasks and MLflow testing. When you use custom modeling functions, all required MLflow actions are performed automatically for you – the default SageMaker AI MLflow App is used automatically, MLflow tests are selected for you and all metrics, parameters, and artifacts are logged for you. From the SageMaker AI Studio model page, you will be able to see the metrics taken from MLflow (as shown in the following image) and continue to view the full metrics within the associated MLflow test.

With MLflow integration it is straightforward to compare models of multiple models. You can use MLflow to visualize performance metrics across tests, identify the best-performing model, and use lists to understand which specific data sets and testers produced that result. This helps you make informed decisions about which models to promote to production based on both quantity and availability metrics.

Getting started with tracking and managing productive AI assets

By bringing these types of customization tools and processes—dataset transformation, tester tracking, model functionality, model deployment—you can turn a fragmented model legacy into a traceable, reproducible, and production-ready workflow with end-to-end automation. This capability is now available in supported AWS Regions. You can access this capability through Amazon SageMaker AI Studio, and the SageMaker python SDK.

To get started:

1. Open Amazon SageMaker AI Studio and navigate to Models part.
2. Customize JumpStart base models to create a model.
3. Navigate to Goods section for managing datasets and testers.
4. Register your first dataset by providing an S3 location and metadata.
5. Create a custom tester using an existing Lambda function or create a new one.
6. Use registered datasets for your optimization tasks—the list is automatically generated.
7. View the model list to see the complete relationship.

For more details, visit the Amazon SageMaker AI documentation.

About the writers

Amit Modi is the product lead for SageMaker AI MLOps, ML Governance, and Inference at AWS. With over a decade of B2B experience, he builds powerful products and teams that drive innovation and deliver value to customers around the world.

Sandeep Raveesh is a GenAI Specialist Solutions Architect at AWS. He works with the customer on their AIOps journey in training models, GenAI applications such as Agents, and measuring GenAI use cases. He also focuses on go-to-market strategies that help build AWS and direct products to solve industry challenges in the productive AI environment. You can connect with Sandeep on LinkedIn to learn about GenAI solutions.