This is the second in a two-part series on the Amazon SageMaker Ground Truth hierarchical labeling workflow and dashboards. In Part 1: Automate multi-modality, parallel data labeling workflows with Amazon SageMaker Ground Truth and AWS Step Functions, we looked at how to create multi-step labeling workflows for hierarchical label taxonomies using AWS Step Functions. In Part 2, we look at how to build dashboards and derive insights for analyzing dataset annotations and worker performance metrics on data lakes generated as output from the complex workflows.
Amazon SageMaker Ground Truth (Ground Truth) is a fully managed data labeling service that makes it easy to build highly accurate training datasets for machine learning (ML). This post introduces a solution that you can use to create customized business intelligence (BI) dashboards using Ground Truth labeling job output data. You can use these dashboards to analyze annotation quality, worker metrics, and more.
In Part 1, we presented a solution to create multiple types of annotations for a single input data object and check annotation quality, using a series of multi-step labeling jobs that run in a parallel, hierarchical fashion using Step Functions. The solution results in high-quality annotations using Ground Truth. The format of these annotations is explained in Output Data, and each takes the form of one or more JSON manifest files in Amazon Simple Storage Service (Amazon S3). You now need a mechanism to dynamically fetch these manifests, publish them on to your analytical datastore, and use them to create meaningful reports in an automated fashion. This allows ML practitioners and data scientists to track annotation progress and quality and allows MLOps and annotation operations teams to gain insights about the annotations and track worker performance. For example, these interested parties may want to see the following reports generated from Ground Truth output data:
In this post, we walk you through the process of generating a data lake for annotations and worker metadata from Ground Truth output data and build visual dashboards on those datasets to gain business insights using Amazon S3, AWS Glue, Amazon Athena, and Amazon QuickSight.
If you completed Part 1 of this series, you can skip the prerequisite and deployment steps and start setting up the AWS Glue ETL job used to process the output data generated from that tutorial. If you didn’t complete Part 1, make sure to complete the prerequisites and deploy the solution, before enabling the AWS Glue workflow.
This post walks you through how to create helpful visualizations for analyzing Ground Truth output data to derive insights into annotations and throughput and efficiency of your own private workers. The walkthrough uses the following AWS services:
In Part 1 of this series, we discuss an architecture pattern that allows you to build a pipeline for orchestrating multi-step data labeling workflows that have workers add different types of annotations to data objects, in parallel, using Ground Truth. In this post, you learn how you can analyze the dataset annotations as well as worker performance. This solution builds data lakes using Ground Truth output data (annotations and worker metadata) and uses these data lakes to derive insights about or analyze the performance of your workers and dataset annotation quality using advanced analytics.
The code for Part 1 and Part 2 is located in the amazon-sagemaker-examples GitHub repo.
The following diagram illustrates this architecture, which is an end-to-end pipeline consisting of two components:
The reporting pipeline is built on the output of the Ground Truth outputs stored in Amazon S3 (referred as the Ground Truth bucket).
The data is processed and the tables are created in the Data Catalog using the following steps:
As shown in the following dashboard examples, you can configure and display the top priority statistics at the top of the dashboard, such as total count of labeled vehicles, quality of labels and frames in a batch, and worker performance metrics. You can create additional visualizations according to your business needs. For more information, see Working with Visual Types in Amazon QuickSight.
The following table includes worker performance summary statistics.
The following dashboard shows several visualizations (from left to right, top to bottom):
If you’re continuing from Part 1 of this series, you can skip this step and move on to enabling the AWS Glue workflow.
If you didn’t completed the demo in Part 1, you need the following resources:
If you didn’t complete the tutorial outlined in Part 1, you can use the sample data provided for this post to create a sample dashboard. If you completed Part 1, you can skip this section and proceed to enabling the AWS Glue workflow.
To launch the resources required to create a sample dashboard with example data, you can launch the stack in AWS Region us-east-1 on the AWS CloudFormation console by choosing Launch Stack:
On the AWS CloudFormation console, choose Next, and modify the parameter for CognitoUserPoolId to identify the user pool associated with your private workforce. You can locate this information on the SageMaker console:
Additionally, enter a prefix to use when naming resources. We use this for creating and managing labeling jobs and worker metrics.
For this post, you can use the default values for the following parameters:
To launch the stack in a different AWS Region, use the instructions found in the README of the GitHub repository.
After you deploy the solution, use the next section to enable an AWS Glue workflow used to generate the BI dashboards.
If you completed Part 1, you launched a CloudFormation stack to create the Ground Truth labeling framework and the annotated MOT17 automotive dataset, using Ground Truth for vehicles and road boundaries and lanes, and audited the frames for quality of the annotations. To convert your data flow into the reporting dashboard set up by Ground Truth Labeling framework, you need to connect the output infrastructure that you previously set up to Athena and QuickSight. Athena can treat data in Amazon S3 as a relational database and allows you to run SQL queries on your data. QuickSight runs those queries on your behalf and creates visualizations of your data.
The following workflow allows Athena to run SQL queries on the example data. Complete the following steps to enable the workflow:
If you don’t want to start the workflow now, you can wait—it automatically runs hourly.
AWS Glue takes some time to spin up its resources during the first run, so allow approximately 30 minutes for the workflow to finish. The completed workflow shows up on the Workflows page.
This pipeline is set up in the reporting.yml file. Currently, the pipeline is run using the AWS Glue workflow using the ScheduledJobTrigger resource with the flag StartOnCreation: false. If you want to run this pipeline on a schedule, switch this flag to true.
All the following metadata and manifest external tables act as base source tables for Ground Truth (SMGT), and they persist values in the same form as they are captured within Ground Truth, with some customization to link the outputed worker ID to identifiable worker information, such as a user name, in the worker metadata. This provides flexibility for auditing and changing analytical needs.
The database ${Prefix}-${AWS::AccountId}-${AWS::Region}-gluedatabase contains four databases, which are surfaced using the AWS Glue workflow. For our demonstration, we use smgt-gluedatabase as the database name. The tables are as follows:
The following screenshot shows the sample output of the tables under the AWS Glue database.
You can use Athena to connect to your S3 data lake and run SQL queries, which QuickSight uses to create visualizations.
If this is your first time using Athena, you need to configure the Athena query result location to the reporting S3 bucket created for the Athena workgroup. For more information, see Specifying a Query Result Location.
SELECT * FROM “smgt-gluedatabase”.”annotations_batch_manifests” limit 10; SELECT * FROM “smgt-gluedatabase”.”worker_metrics_processed_worker_metrics” limit 10; SELECT * FROM “smgt-gluedatabase”.”output_manifest_videoobjecttracking” limit 10; SELECT * FROM “smgt-gluedatabase”.”output_manifest_videoobjecttrackingaudit” limit 10;
You must have at least one Ground Truth job completed to generate these tables.
The following screenshot shows our output.
You’re now ready to visualize your data in QuickSight.
In this section, you update permissions in your QuickSight account to provide access to the S3 reporting buckets. For more information, see Accessing Data Sources. You also import the data from Athena to SPICE so that QuickSight can display it.
A list of available AWS services is displayed.
Create a new dataset using Athena as the source.
SELECT *, cardinality(ans.trackingannotations.framedata.entries) as tasks FROM “smgt-gluedatabase”.”worker_metrics_processed_worker_metrics”, unnest(answercontent) as t(ans);
The following code creates a label-level dataset for vehicles:
SELECT job_name,each_ann.height,each_ann.width,each_ann.top,each_ann.”left”,each_ann.”label-category-attributes”.moving,each_ann.”label-category-attributes”.vehicle_type,each_ann.”label-category-attributes”.audit,each_ann.”object-name”,each_ann from (SELECT ann.annotations, partition_1 as job_name FROM “smgt-gluedatabase”.”annotations_batch_manifests”, unnest(“tracking-annotations”) as t(ann) where cardinality(ann.annotations) != 0) as data, unnest(data.annotations) as t(each_ann);
The following code creates a frame-level dataset for vehicles:
SELECT ann.”frame-no”,ann.frame,ann.”frame-attributes”.”number_of_vehicles”,ann.”frame-attributes”.”quality_of_the_frame”,ann.annotations, cardinality(ann.annotations) as num_labels, partition_1 as job_name, ann FROM “smgt-gluedatabase”.”annotations_batch_manifests”, unnest(“tracking-annotations”) as t(ann) where cardinality(ann.annotations) != 0
Next, you create a new analysis that imports the data from Athena to SPICE so that QuickSight can display it.
QuickSight enables you to visualize your tabular data. For more information, see Creating an Amazon QuickSight Visual.
The following table summarizes several useful worker metric graphs that you can add to your dashboard.
Table Name |
Graph Type | Field Wells Value | Field Wells X-axis | Field Wells Row | Field Wells Columns |
Group/Color |
Total time spent labeling by a worker | Vertical stacked bar chart | timespentinseconds(Sum) | user name | modality | ||
Total time spent by modality | Autograph | timespentinseconds(Sum) | modality | |||
Worker metrics table | Table | timespentinSeconds(sum) tasks(sum) timespentinseconds (Max) timespentinseconds (Min) Average Time Taken Per Video (Average) |
user name |
You can add these tables to your QuickSight dashboard by creating a visual and customizing according to your requirements.
The follow are best practices for using the tables:
For more information about how to create visuals, calculated fields, parameters, controls, and visual tables, see Dashboard Building 101.
The following example visualization uses the Amazon Cognito worker sub IDs to identify worker metadata (such as email addresses). If you didn’t complete Part 1 and are using the example data provided for this post, these sub IDs aren’t associated with worker metadata in Amazon Cognito, so the sub ID appears in place of user names in the table. To learn more about using worker sub IDs with worker information, see Tracking the throughput of your private labeling team through Amazon SageMaker Ground Truth.
The following table summarizes several useful annotation graphs that you can add to your dashboard.
Table Name |
Graph Type | Field Wells Value | Field Wells Y-axis | Field Wells Row | Field Wells Columns |
Group/Color |
Number of vehicles | Pie Chart | vehicle_type (Count) | vehicle_type | |||
Annotation level quality | Donut Chart | audit | ||||
Frame level quality | Donut Chart | quality_of_the_frame | ||||
Number of parked vehicles vs vehicles in motion | Donut Chart | moving | ||||
Maximum number of vehicles in a frame | Horizontal Bar Chart | number_of_vechicles (Count) | ||||
Quality of the frame per Job | Table | quality_of_the_frame (Count) | job_name | quality_of_the_frame | ||
Quality of the labels per Job | Table | audit (Count) | job_name | audit |
The following screenshot shows a sample dashboard for these annotation reports.
To download your worker metrics and annotation reports as a CSV file, choose the respective sheet. In the Options section, choose Menu options and then choose Export to CSV.
For more information, see Exporting Data.
To refresh your dashboard every hour, set the SPICE refresh schedule to be 1 hour for newly created datasets. For instructions, see Refreshing a Dataset on a Schedule.
We show the sample QuickSight dashboards when data is ingested from the Ground Truth output data in the preceding sections.
If you want to build dashboards on your current Ground Truth output data directories, you can make customizations:
To remove all resources created throughout this process and prevent additional costs, complete the following steps:
This two-part series provides you with a reference architecture to build an advanced data labeling workflow comprised of a multi-step data labeling pipeline, adjustment jobs, and data lakes for corresponding dataset annotations and worker metrics as well as updated dashboards.
In this post, you learned how to generate data lakes for annotations and worker metadata from Ground Truth output data generated from Part 1 using Ground Truth, Amazon S3, and AWS Glue. Then we discussed how to build visual dashboards for your annotation and worker metadata reports on those data lakes to derive business insights using Athena and QuickSight.
To learn more about automatic model building, selection, and deployment of custom classification models, refer to Automate multi-modality, parallel data labeling workflows with Amazon SageMaker Ground Truth and AWS Step Functions.
Try out the notebook and customize it for your label configuration by adding additional jobs or audit steps, or by modifying the data modality of the jobs. Further customization could include, but is not limited, to:
This solution is built using serverless technologies on top of AWS Glue and Amazon S3, which makes it highly customizable and applicable for a wide variety of applications. We encourage you to extend this pipeline to your data analytics and visualization use cases—there are many more transformations in AWS Glue, capabilities to build complex queries using Athena, and prebuilt visuals in QuickSight to explore.
Happy building!
Vidya Sagar Ravipati is a Deep Learning Architect at the Amazon ML Solutions Lab, where he leverages his vast experience in large-scale distributed systems and his passion for machine learning to help AWS customers across different industry verticals accelerate their AI and cloud adoption. Previously, he was a Machine Learning Engineer in Connectivity Services at Amazon who helped to build personalization and predictive maintenance platforms.
Gaurav Rele is a Data Scientist at the Amazon ML Solution Lab, where he works with AWS customers across different verticals to accelerate their use of machine learning and AWS Cloud services to solve their business challenges.
Talia Chopra is a Technical Writer in AWS specializing in machine learning and artificial intelligence. She works with multiple teams in AWS to create technical documentation and tutorials for customers using Amazon SageMaker, MxNet, and AutoGluon.