This is a guest post by Jihye Park, a Data Scientist at MUSINSA.
MUSINSA is one of the largest online fashion platforms in South Korea, serving 8.4M customers and selling 6,000 fashion brands. Our monthly user traffic reaches 4M, and over 90% of our demographics consist of teens and young adults who are sensitive to fashion trends. MUSINSA is a trend-setting platform leader in the country, leading with massive amounts of data.
The MUSINSA Data Solution Team engages in everything related to data collected from the MUSINSA Store. We do full stack development from log collection to data modeling and model serving. We develop various data-based products, including the Live Product Recommendation Service on our app’s main page and the Keyword Highlighting Service that detects and highlights words such as ‘size’ or ‘satisfaction level’ from text reviews.
The quality and quantity of customer reviews are critical for ecommerce businesses, as customers make purchase decisions without seeing the products in person. We give credits to those who write image reviews on the products they purchased (that is, reviews with photos of the products or photos of them wearing/using the products) to enhance customer experience and increase the purchase conversion rate. To determine if the submitted photos met our criteria for credits, all of the photos are inspected individually by humans. For example, our criteria states that a “Style Review” should contain photos featuring the whole body of a person wearing/using the product while a “Product Review” should provide a full shot of the product. The following images show examples of a Product Review and a Style Review. Uploaders’ consent has been granted for use of the photos.
Examples of Product Review. |
Examples of Style Review. |
Over 20,000 photos are uploaded daily to the MUSINSA Store platform that require inspection. The inspection process classifies images as ‘package’, ‘product’, ‘full-length’, or ‘half-length’. The image inspection process is completely manual, so it was extremely time consuming and classifications are often done differently by different individuals, even with the guidelines. Faced with this challenge, we used Amazon SageMaker to automate this task.
Amazon SageMaker is a fully managed service for building, training, and deploying machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows. It let us quickly implement the automated image inspection service with good results.
We will go into detail about how we addressed our problems using ML models and used Amazon SageMaker along the way.
The first step toward automating the Image Review Inspection process was to manually label images, thereby matching them to the appropriate categories and inspection criteria. For example, we classified images as a “full body shot,” “upper body shot,” “packaging shot,” “product shot,” etc. In the case of a Product Review, credits were given only for a product shot image. Likewise, in the case of a Style Review, credits were given for a full body shot.
As for image classification, we largely depended on a pre-trained convolutional neural network (CNN) model due to the sheer volume of input images required to train our model. While defining and categorizing meaningful features from images are both critical to training a model, an image can have a limitless number of features. Therefore, using the CNN model made the most sense, and we pre-trained our model with 10,000+ ImageNet datasets, then we used transfer learning. This meant that our model could be trained more effectively with our image labels later.
However, transfer learning had its own limitations, because a model must be newly trained on higher layers. This means that it constantly required input images. On the other hand, this method performed well and required fewer input images when trained on entire layers. It easily identified features from images from these layers because it had already been trained with a massive amount of data. At MUSINSA, our entire infrastructure runs on AWS, and we are storing customer-uploaded photos in Amazon Simple Storage Service (S3). We categorized these images into different folders based on the labels we defined, and we used Amazon SageMaker Ground Truth for the following reasons:
In summary, categorizing 10,000 images required 22 inspectors five days and cost $980.
We needed to classify review images as full body shots, upper body shots, package shots, product shots, and products into applicable categories. To accomplish our goals, we considered two models: the ResNet-based SageMaker built-in model and the Tensorflow-based MobileNet. We tested both on the same test datasets and found that the SageMaker built-in model was more accurate, with a 0.98 F1 score vs 0.88 from the TensorFlow model. Therefore, we decided on the SageMaker built-in model.
The SageMaker Studio-based model training process was as follows:
SageMaker made it straightforward to train the model with just one click and without worrying about provisioning and managing a fleet of servers for training.
For hyperparameter turning, we employed grid search to determine the optimal values of hyperparameters, as the number of training layers (num_layers) and training cycles (epochs) during transfer learning had affected our classification model accuracy.
epochs_list = [5, 10, 15] num_layers_list = [18, 34, 50] from sagemaker.analytics import TrainingJobAnalytics metric_df = pd.DataFrame() for i in range(len(epochs_list)): for j in range(len(num_layers_list)): # hyperparameter settings ic.set_hyperparameters(num_layers=num_layers_list[j], use_pretrained_model=1, image_shape = “3,256,256”, num_classes=9, num_training_samples=50399, mini_batch_size=128, epochs=epochs_list[i], learning_rate=0.01, precision_dtype=’float32′) ic.fit(inputs=data_channels, logs=True) latest_job_name = ic.latest_training_job.job_name job_metric = TrainingJobAnalytics(training_job_name=latest_job_name).dataframe() job_metric[‘epochs’] = epochs_list[i] job_metric[‘num_layers’] = num_layers_list[j] metric_df = pd.concat([metric_df, job_metric])
The image classification model we built required ML workflows to determine if a review image was qualified for credits. We established workflows with the following four steps.
We are using Apache Airflow to manage data product workflows. It is a workflow scheduling and monitoring platform developed by Airbnb known for simple and intuitive web UI graphs. It supports Amazon SageMaker, so it easily migrates the code developed with SageMaker Studio to Apache Airflow. There are two ways to run SageMaker jobs on Apache Airflow:
def transform(dt, bucket, training_job, **kwargs): estimator = sagemaker.estimator.Estimator.attach(training_job) transformer = estimator.transformer(instance_count=1, instance_type=’ml.m4.xlarge’, output_path=f’s3://{bucket}/…/dt={dt}’, max_payload=1) transformer.transform(data=f’s3://{bucket}/…/dt={dt}’, data_type=’S3Prefix’, content_type=’application/x-image’, split_type=’None’) transformer.wait() … transform_op = PythonOperator( task_id=’transform’, dag=dag, provide_context=True, python_callable=transform, op_kwargs={“dt”: dt, “bucket”: bucket, “training_job”: training_job})
The second option let us maintain our existing Python codes that we already had on SageMaker Studio, and it didn’t require us to learn new grammars for Amazon SageMaker Operators.
However, we went through some trial and error, as it was our first time integrating Apache Airflow with Amazon SageMaker. The lessons we learned were:
By automating review image inspection processes, we gained the following business outcomes:
We gained the following benefits specifically by using Amazon SageMaker to automate the image inspection process:
Going forward, we plan to automate not only model serving but also model training through automatic batches. We want our model to identify the optimal hyperparameters automatically when new labels or images are added. In addition, we will continue improving the performance of our model, namely recalls and precision, based on the previously mentioned automated training method. We will increase our model coverage so that it can inspect more review images, reduce more costs, and achieve higher accuracies, which will all lead to higher customer satisfaction.
For more information about how to use Amazon SageMaker to solve your business problems using ML, visit the product webpage. And, as always, stay up to date with the latest AWS Machine Learning News here.
The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.
Jihye Park is a Data Scientist at MUSINSA who is responsible for data analysis and modeling. She loves working with ubiquitous data such as ecommerce. Her main role is data modeling but she has interests in data engineering too.
Sungmin Kim is a Sr. Solutions Architect at Amazon Web Services. He works with startups to architect, design, automated, and build solutions on AWS for their business needs. He specializes in AI/ML and Analytics.