Launched at AWS re:Invent 2020, Amazon SageMaker Pipelines is the first purpose-built, easy-to-use continuous integration and continuous delivery (CI/CD) service for machine learning (ML). With Pipelines, you can create, automate, and manage end-to-end ML workflows at scale.
You can integrate Pipelines with existing CI/CD tooling. This includes integration with existing source control systems such as GitHub, GitHub Enterprise, and Bitbucket. This new capability also allows you to utilize existing installations of Jenkins for orchestrating your ML pipelines. Before this new feature, Amazon SageMaker projects and pipelines were optimized for use with AWS Developer Tools including AWS CodePipeline, AWS CodeCommit, and AWS CodeBuild. This new capability allows you to take advantage of Pipelines while still using existing skill sets and tooling when building your ML CI/CD pipelines.
With the newly added MLOps project templates, you can choose between the following options:
The new template options are now available via the SDK or within the Amazon SageMaker Studio IDE, as shown in the following screenshot.
In this post, we walk through an example using GitHub and Jenkins to demonstrate these new capabilities. You can perform equivalent steps using GitHub Enterprise or Bitbucket as your source code repository. The MLOps project template specifically creates a CI/CD pipeline using Jenkins to build a model using a SageMaker pipeline. The resulting trained ML model is deployed from the model registry to staging and production environments.
The following are prerequisites to completing the steps in this post:
We also assume some familiarity with Jenkins. For general information on Jenkins, we recommend reading the Jenkins Handbook.
In the following sections, we cover the one-time setup tasks and the steps required when building new pipelines using the new SageMaker MLOps project templates to build out the following high-level architecture (click on image to expand).
The model build pipeline is triggered based on changes to the model build GitHub repository based on Jenkins polling the source repository every minute. The model deploy pipeline can be triggered based on changes to the model deploy code in GitHub or when a new model version is approved in the SageMaker Model Registry.
The one-time setup tasks include:
We then use the new MLOps project template for third-party GitHub and Jenkins to provision and configure the following resources, which are also discussed in more detail later in this post:
All of the provisioning and configuration required to set up the end-to-end CI/CD pipeline using these resources is automatically performed by SageMaker projects.
Now that we’ve covered how the new feature works, let’s walk through the one-time setup tasks followed by using the new templates.
The tasks in this section are required as part of the one-time setup activities that must be performed for each AWS Region where you use the new SageMaker MLOps project templates. The steps to create a GitHub connection and an AWS Identity and Access Management (IAM) user for Jenkins could be incorporated into a CloudFormation template for repeatability. For this post, we explicitly define the steps.
In this step, you connect to your GitHub repositories using AWS Developer Tools and, more specifically, AWS CodeStar connections. The SageMaker project uses this connection to connect to your source code repositories.
A list of all the GitHub personal accounts and organizations you have access to is displayed.
When the app is installed, you’re redirected to the Connect to GitHub page and the installation ID is automatically populated.
You use the ARN as a parameter in the project creation step.
In this step, you ensure that several software dependencies are in place on the Jenkins server. If you don’t have an existing Jenkins server or need to create one for testing, you can install Jenkins.
On Unix or Mac, enter the following code:
sudo yum install python3-pip
On Ubuntu, enter the following code:
sudo apt install python3-pip
In this step, you create an IAM user and permissions policy that allows for programmatic access to Amazon S3, SageMaker, and AWS CloudFormation. This IAM user is used by your Jenkins server to access the AWS resources needed to configure the integration with SageMaker projects and your Jenkins server. After this user is created, you configure the same on the Jenkins server using the IAM user credentials.
{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “s3:CreateBucket”, “s3:PutObject” ], “Resource”: [ “arn:aws:s3:::sagemaker-*” ] }, { “Effect”: “Allow”, “Action”: [ “iam:PassRole” ], “Resource”: [ “arn:aws:iam::*:role/service-role/AmazonSageMakerServiceCatalogProductsUseRole” ] }, { “Effect”: “Allow”, “Action”: [ “sagemaker:CreatePipeline”, “sagemaker:DescribePipeline”, “sagemaker:DescribePipelineExecution”, “sagemaker:ListPipelineExecutionSteps”, “sagemaker:StartPipelineExecution”, “sagemaker:UpdatePipeline”, “sagemaker:ListModelPackages”, “sagemaker:ListTags”, “sagemaker:AddTags”, “sagemaker:DeleteTags”, “sagemaker:CreateModel”, “sagemaker:CreateEndpointConfig”, “sagemaker:CreateEndpoint”, “sagemaker:DeleteModel”, “sagemaker:DeleteEndpointConfig”, “sagemaker:DeleteEndpoint”, “sagemaker:DescribeEndpoint”, “sagemaker:DescribeModel”, “sagemaker:DescribeEndpointConfig”, “sagemaker:UpdateEndpoint” ], “Resource”: “arn:aws:sagemaker:${AWS::Region}:${AWS::AccountId}:*” }, { “Effect”: “Allow”, “Action”: [ “cloudformation:CreateStack”, “cloudformation:DescribeStacks”, “cloudformation:UpdateStack”, “cloudformation:DeleteStack” ], “Resource”: “arn:aws:cloudformation:*:*:stack/sagemaker-*” } ] }
We now need to create a user that the policy is attached to.
You need the access key ID and secret key for Jenkins to be able to create and run the CI/CD pipeline. The secret key is only displayed one time, so make sure to save both values in a secure place.
In this step, you configure the AWS credentials for the Jenkins IAM user on your Jenkins server. To do this, you need to sign in to your Jenkins server with administrative credentials. The credentials are stored in the Jenkins Credential Store.
Your new credentials are now listed under Global credentials.
In this step, you configure the trigger to run your Jenkins model deployment pipeline whenever a new model version gets registered into a model package group in the SageMaker Model Registry. To do this, you create an API token for communication with your Jenkins server. Then you run a CloudFormation template from your AWS account that sets up a new rule in EventBridge to monitor the approval status of a model package registered in the SageMaker Model Registry. We use the model registry to catalog models and metadata about those models, as well as manage the approval status and model deployment pipelines. The CloudFormation template also creates a Lambda function that is the event target when a new model gets registered. This function gets the Jenkins API user token credentials from AWS Secrets Manager and uses that to trigger the pipeline remotely based on the trigger, as shown in the following diagram (click on the image to expand).
First, you need to create an API token for the Jenkins user.
Next, you create the trigger and Lambda function. To do this, you need the provided CloudFormation template, model_trigger.yml. The template takes three parameters as input:
You can download and launch the CloudFormation template via the AWS CloudFormation Console, the AWS Command Line Interface (AWS CLI), or the SDK, or by simply choosing the following launch button:
This completes the one-time setup required to use the new MLOps SageMaker project templates for each Region. Depending on your organizational structure and roles across the ML development lifecycle, these one-time setup steps may need to be performed by your DevOps, MLOps, or system administrators.
We now move on to the steps for creating SageMaker projects using the new MLOps project template from SageMaker Studio.
In this section, we cover how to use one of the two new MLOps project templates released that allow you to utilize Jenkins as your orchestrator. First, we create a new SageMaker project using one of the new templates. Then we use the generated Jenkins pipeline code to create the Jenkins pipeline.
To create your SageMaker project, complete the following steps:
You need to provide several parameters to configure the source code repositories for your model build and model deploy code.
The seed code includes model build code for the abalone use case that is common to SageMaker projects; however, when this is enabled, a new /jenkins folder with Jenkins pipeline code is also seeded.
It’s recommended to allow SageMaker projects to seed your repositories with the code to ensure proper structure and for automatic generation of the Jenkins DSL pipeline code. If you don’t choose this option, you need to create your own Jenkins DSL pipeline code. You can then modify the seed code specific to your model based on your use case.
As we mentioned earlier, the seed code includes the model deploy code for the abalone use case that is common to SageMaker projects; however, when this is enabled, a /jenkins folder with Jenkins pipeline code is also seeded.
A message appears indicating that SageMaker is provisioning and configuring the resources.
When the project is complete, you receive a successful message, and your project is now listed on the Projects list.
You now have seed code in your abalone-model-build and abalone-model-deploy GitHub repositories. You also have the /jenkins folders containing the Jenkins DSL to create your Jenkins pipeline.
After you create the SageMaker project with seed code enabled, the code needed to create a Jenkins pipeline is automatically generated. Let’s review the code generated and push to the abalone-model-build and abalone-model-deploy GitHub repositories.
The model build pipeline contains the following:
The model deploy pipeline contains the following:
In this step, we create the Jenkins pipeline using the DSL generated in the seed code created through the SageMaker project in the previous step.
You must provide the following information for the AWS credentials that are used by your Jenkins pipeline to integrate with AWS.
seed_job.groovy was automatically generated by your SageMaker project and pushed to your GitHub repository when seeding was indicated.
Next, we want to run our Jenkins job to create the Jenkins pipeline.
The first run of the pipeline fails with an error that the script is not approved. Jenkins implements security controls to ensure only approved user-provided groovy scripts can be run (for more information, see In-process Script Approval). As a result, we need to approve the script before running the build again.
You should see a message that a script is pending approval.
This time, the job should run successfully and create a new modelbuild pipeline.
This is the pipeline generated by the Jenkins DSL code that was seeded in your GitHub repository. This is the actual model building pipeline.
You still have visibility to your model build pipeline, but the orchestration for the CI/CD pipeline steps is performed by Jenkins.
If a data scientist wants to update any of the model build code, they can clone the repository to their Studio environment by choosing clone repo. When new code is committed and pushed to the GitHub repository, the Jenkins model build pipeline is automatically triggered.
In this step, we perform the same steps as we did with the model build pipeline to create a model deploy pipeline, using the model deploy GitHub repo.
You can now see a new pipeline called sagemaker-jenkings-btd-1-p-
The first time this pipeline builds, it fails. Similar to the previous steps, you need to approve the script and rebuild the pipeline.
After the two pipelines are created, two additional pipelines appear in Jenkins that are associated with the SageMaker project.
The model deploy pipeline fails because the first time it runs, there are no approved models in the model registry.
When you navigate to the model registry, you can see a model that has been trained and registered by the model build pipeline. You can approve the model by updating its status, which triggers the deploy pipeline.
You can see the deploy pipeline running and the model is deployed to a staging environment.
After the model is deployed to staging, a manual approval option is available to deploy the model into a production environment
On the SageMaker console, the endpoint deployed by Jenkins is also visible.
After you approve the Jenkins pipeline, a model is deployed to a production environment and is visible on the SageMaker console.
In this post, we walked through one of the new SageMaker MLOps project templates that you can use to build and configure a CI/CD pipeline that takes advantage of SageMaker features for model building, training, and deployment while still using your existing tooling and skillsets. For our use case, we focused on using GitHub and Jenkins, but you can also use GitHub Enterprise or Bitbucket depending on your needs. You can also utilize the other new template to combine your choice of source code repository (GitHub, GitHub Enterprise, or Bitbucket) with CodePipeline. Try it out and let us know if you have any questions in the comments section!
Shelbee Eigenbrode is a Principal AI and Machine Learning Specialist Solutions Architect at Amazon Web Services (AWS). She holds 6 AWS certifications and has been in technology for 23 years spanning multiple industries, technologies, and roles. She is currently focusing on combining her DevOps and ML background to deliver and manage ML workloads at scale. With over 35 patents granted across various technology domains, she has a passion for continuous innovation and using data to drive business outcomes. Shelbee co-founded the Denver chapter of Women in Big Data.
Saumitra Vikram is a Software Developer on the Amazon SageMaker team and is based in Chennai, India. Outside of work, he loves spending time running, trekking and motor bike riding through the Himalayas.
Venkatesh Krishnan is a Principal Product Manager – Technical for Amazon SageMaker in AWS. He is the product owner for a portfolio of services in the MLOps space including SageMaker Pipelines, Model Registry, Projects, and Experiments. Earlier he was the Head of Product, Integrations and the lead product manager for Amazon AppFlow, a new AWS service that he helped build from the ground up. Before joining Amazon in 2018, Venkatesh served in various research, engineering, and product roles at Qualcomm, Inc. He holds a PhD in Electrical and Computer Engineering from Georgia Tech and an MBA from UCLA’s Anderson School of Management.
Kirit Thadaka is an ML Solutions Architect working in the SageMaker Service SA team. Prior to joining AWS, Kirit spent time working in early stage AI startups followed by some time in consulting in various roles in AI research, MLOps, and technical leadership.