Computer vision for creative optimization

6 min readNov 5, 2022

KPI maximization through image analysis

INTRODUCTION

Our client is Ad-tech company— an online mobile ad business, which provides services as Design an interactive Ad, serving creatives and optimizing the creative designing process to its clients. The companyhas been running a vast number of advertisements with respective creatives, but these creatives were made based on the experience of designers and company needs. It also plans to change this predicament by developing an algorithm that helps it optimize its creatives based on campaign performance data. Here is where we are supposed to come into play, we are tasked to develop a deep learning-based computer vision algorithm that segments objects from creative assets and relates them to KPI parameters of the corresponding campaigns.

The Dataset Archive provided in an s3 bucket is composed of an ‘Assets’ folder and a ‘performance_data’ CSV file. As the names suggest the folder contains the asset image used to build the creatives and the CSV file contains the performance values for each creative. Each folder within the ‘Assets’ folder is a ‘game_id’ string value and its reference performance is available in the CSV file. The following are column values from the ‘performace_data.csv’ file.

game_id — Represents a unique identifier of a creative for the performance values
ER — Represents the engagement rate score of the creative
CTR — Represents the click-through rate score of the creative

Why dynamic creative optimization?

Dynamic creative optimization (DCO) is a supercharged form of dynamic creative that uses artificial intelligence (AI) and machine learning technology to create hyper-personalized experiences for viewers. It’s an automated process that leverages existing customer data and other connected data sources plus real-time testing and analytics to select the most effective combination of creative elements for each viewer.

There are many benefits of incorporating dynamic creative optimization as part of your digital marketing strategy. The top four benefits are personalization, automation, real-time response and improved ROI. These are achieved by applying machine learning to enable advertisers gain the following five key flairs.

Capitalize on unexpected insights
Improve ad creative
Boost contextual relevance
Target more defined segments
Bid more strategically

State of the art image segmentation algorithms

Image segmentation allows us to decompose an image into meaningful parts and understand a scene at a more granular level. The tasks of segmentation can be broken down into two broad categories: semantic segmentation and instance segmentation. In semantic segmentation, each pixel belongs to a particular class (think classification on a pixel level). In the image above, for example, those classes were bus, car, tree, building, etc. Any pixel belonging to any car is assigned to the same “car” class. Instance segmentation goes one step further and separates distinct objects belonging to the same class. For example, if there were two cars in the image above, each car would be assigned the “car” label, but would be given a distinct color because they are different instances of the class. The following are the most common algorithms used for image segmentation.

Otsu’s Image segmentation (Threshold-based segmentation)

It comes under threshold-based segmentation. In Otsu’s Segmentation, the input image is first processed, and then we try to obtain the histogram of the image, which will show the distribution of pixels in the image. Here we focus on peak value. The next step is to compute the threshold value and compare it with the image pixels. Set the pixel to white; if they are greater than the threshold else, set it to black. Thus, it performs automatic thresholding. This method is not suitable for noisy images. Applications include scanning documents, recognizing patterns.

Edge-based image segmentation algorithms

In this method, an edge filter is applied to the image. The image pixels are then classified as edge or non-edge depending on the filter output. Edge detection helps to remove unwanted and unnecessary information from the image. It includes 2 steps- Edge detection and edge linking. The different types of edge detection operators are:

Robert operator: It highlights regions of high spatial frequency
Prewitt operator: It is used for detecting vertical and horizontal edges in images
Sobel operator: It is used for detecting vertical and horizontal edges in images.

Region-based image segmentation algorithm

Here we grow the regions recursively by including the neighbor pixels which are similar and connected to that pixel, and we will use similarity measures for regions with homogeneous grey levels. The process is iterated in the same way as any general data clustering algorithm. Some of the advantages of this method include faster and easier computation, better performance, etc.

Clustering-based image segmentation algorithms

These are methods like fuzzy c-means, k-means, improved k-means, etc. Here we will take each point as a separate cluster and merge two clusters with the minimum inter-cluster distance. Repeat this step until clustering is satisfactory. Another approach is using K means algorithm where we select K points and assign them a cluster center by calculating the mean. Now, allocate other points nearest to the particular centers and form clusters. Repeat unlit the cluster centers don’t change.

Neural networks for image segmentation

Segmentation may require large neural networks. CNN is most widely used for object detection, pattern recognition etc. It can even process large datasets. They are used for image segmentation due to their high accuracy. It works as a hierarchical model with various layers and results in a segmented image as the output.

A great tool for image segmentation; Yolo 7

It is one of the image segmentation tools, YOLO stands for You Only Look Once. It is a Deep Learning model used for detection on images and videos. The first version of YOLO was released in 2016. Since then, frequent updates are made with the latest improvements: faster computation, better accuracy. Refer this on how to use it.

Methodology

Access the data from the s3 bucket and study the folder structure. Also understanding the performance data so that to understand how to relate it to the assets directory. We can do this by using the game_id. We can also append extracted starting and end frames by using game_id = folder_name, which is also a sub-string of the preview link.
Then proceed to feature extraction. Extract important features which could be very useful in KPI determination from image files. These features are shared in the results section of this blog.
Finally after appending this extracted features to the performance data, and applying some pre-processing like dropping non-English text and normalizing, labeling and scaling, we fed the data to machine learning model, trained the model and saved in in the models folder.

Results

1. features extracted and their importance

2. Machine learning results and evaluation metrics of the applied algorithms

3. Correlation matrix among extracted features

Conclusion

This project provided us with very useful skills in image processing and computer vision. It exposed us to the useful libraries and packages. This should be a springboard project to lift up our skill in image processing. We worked with Deep Learning frameworks e.g. pyTorch and Tensorflow, optimized image segmentation deep learning architectures, using CV public models and APIs and finally ML modelling with KPIs and ML artifacts by formulating and designing test and training strategies. We were able to extract interesting features and present them over a streamlit dashboard. So we overall completed the assigned tasks and thereby the project.

DEPLOYMENT:

https://amanuel3065-creative-image-optimization-app-yu2h7h.streamlit.app/

Future Work

In the near future, we would like to take more details into account while fitting our model. This will give us a lot more room when choosing which feature is more deciding. We also plan to build a more interactive and precise dashboard to present better visualizations and predict indicators by just taking a creative’s link or an asset directory from users.