Machine Studying Incidents in AdTech | by Ben Weber | Jan, 2025

One of many largest challenges I encountered in my profession as a knowledge scientist was migrating the core algorithms in a cellular AdTech platform from traditional machine studying fashions to deep studying. I labored on a Demand Facet Platform (DSP) for person acquisition, the place the function of the ML fashions is to foretell if exhibiting advert impressions to a tool will consequence within the person clicking on the advert and putting in a cellular app. For a fast hands-on overview of the clicking prediction drawback, please try my previous submit.

Whereas we have been in a position to rapidly get to a state the place the offline metrics for the deep studying fashions have been aggressive with logistic regression fashions, it took awhile to get the deep studying fashions working easily in manufacturing, and we encountered many incidents alongside the best way. We have been in a position to begin with small-scale exams utilizing Keras for mannequin coaching and Vertex AI for managed TensorFlow serving, and ran experiments to check iterations of our deep studying fashions with our champion logistic regression fashions. We have been ultimately in a position to get the deep studying fashions to outperform the traditional ML fashions in manufacturing, and modernize our ML platform for person acquisition.

When working with machine studying fashions on the core of a posh system, there are going to be conditions the place issues are going to go off the rails and it’s vital to have the ability to rapidly recuperate and study from these incidents. Throughout my time at Twitch, we used the 5 W’s strategy to writing postmortems for incidents. The thought is to determine “what” went mistaken, “when” and “the place” it occurred, “who” was concerned, and “why” an issue resulted. The comply with up is to then set up learn how to keep away from the sort of incident sooner or later and to arrange guardrails to stop comparable points. The purpose is to construct a increasingly more sturdy system over time.

In one in every of my previous roles in AdTech, we bumped into a number of points when migrating from traditional ML fashions to deep studying. We ultimately bought to a state the place we had a sturdy pipeline for coaching, validating, and deploying fashions that improved upon our traditional fashions, however we bumped into incidents throughout this course of. On this submit we’ll cowl 8 of the incidents that occurred and describe the next steps we took for incident administration:

1. What was the difficulty?
2. How was it discovered?
3. How was it fastened?
4. What did we study?

We recognized a wide range of root causes, however typically aligned on comparable options when making our mannequin pipelines extra sturdy. I hope sharing particulars about these incidents gives some steerage on what can go mistaken when utilizing deep studying in manufacturing.

Incident 1: Untrained Embeddings

What was the difficulty?
We discovered that most of the fashions that we deployed, comparable to predicting click on and set up conversion, have been poorly calibrated. This meant that the expected worth of conversion by the mannequin was a lot larger than the precise conversion that we noticed for impressions that we served. After drilling down additional, we discovered that the miscalibration was worse on categorical options the place we had sparse coaching information. Finally we found that we had embedding layers in our set up mannequin the place we had no coaching information out there for a few of the vocabulary entries. What this meant is that when becoming the mannequin, we weren’t making any updates to those entries, and the coefficients remained set to their randomized initialized weights. We known as this incident “Untrained Embeddings”, as a result of we had embedding layers the place a few of the layer weights by no means modified throughout mannequin coaching.

How was it discovered?
We largely found this concern by instinct after reviewing our fashions and information units. We used the identical vocabularies for categorical options throughout two fashions, and the set up mannequin information set was smaller than the clicking mannequin information set. This meant that a few of the vocabulary entries that have been high quality to make use of for the clicking mannequin have been problematic for the set up mannequin, as a result of a few of the vocab entries didn’t have coaching examples within the smaller information set. We confirmed that this was the difficulty by evaluating the weights within the embedding layers earlier than and after coaching, and discovering {that a} subset of the weights have been unchanged after becoming the mannequin. As a result of we randomly initialized the weights in our Keras fashions, this led to points with the mannequin calibration on reside information.

How was it fastened?
We first restricted the dimensions of our vocabularies used for categorical options to scale back the probability of this concern occurring. The second change we made was setting the weights to 0 for any embedding layers entries the place the weights have been unchanged throughout coaching. Long run, we moved away from reusing vocabularies throughout totally different prediction duties.

What did we study?
We found that this was one of many points that was resulting in mannequin instability, the place fashions with comparable efficiency on offline metrics would have noticeably totally different efficiency when deployed to manufacturing. We ended up constructing extra tooling to check mannequin weights throughout coaching runs as a part of our mannequin validation pipeline.

Incident 2: Padding Subject with Batching for TensorFlow Serving

What was the difficulty?
We migrated from Vertex AI for mannequin serving to an in-house deployment of TensorFlow serving, to take care of a few of the tail-end latency points that we have been encountering with Vertex on the time. When making this modification, we bumped into a difficulty with learn how to take care of sparse tensors when enabling batching for TensorFlow serving. Our fashions contained sparse tensors for options, such because the checklist of identified apps put in on a tool, that could possibly be empty. Once we enabled batching when serving on Vertex AI, we have been in a position to make use of empty arrays with out concern, however for our in-house mannequin serving we bought error responses when utilizing batching and passing empty arrays. We ended up passing “[0]” values as a substitute of “[ ]” tensor values to keep away from this concern, however this once more resulted in poorly calibrated fashions. The core concern is that “0” referred to a selected app fairly than getting used for out-of-vocab (OOV). We have been introducing a function parity concern to our fashions, as a result of we solely made this modification for mannequin serving and never for mannequin coaching.

How was it discovered?
As soon as we recognized the change that had been made, it was easy to exhibit that this padding strategy was problematic. We took information with an empty tensor and adjusted the worth from “[]” to “[0]” whereas protecting all the different tensor values fixed, and confirmed that this modification resulted in numerous prediction values. This made sense, as a result of we have been altering the tensor information to assert that an app was put in on the gadget the place that was not really the case.

How was it fastened?
Our preliminary repair was to vary the mannequin coaching pipeline to carry out the identical logic that we carried out for mannequin serving, the place we change empty arrays with “[0]”, however this didn’t fully tackle this concern. We later modified the vocab vary from [0, n-1] to [0, n], the place 0 had no that means and was added to each tensor. This meant that each sparse tensor had no less than 1 worth and we have been in a position to make use of batching with our sparse tensor setup.

What did we study?
This concern largely got here up because of totally different threads of labor on the mannequin coaching and mannequin serving pipelines, and lack of coordination. As soon as we recognized the variations between the coaching and serving pipelines, it was apparent that this discrepancy might trigger points. We labored to enhance on this incident by together with information scientists as reviewers on pull requests on the manufacturing pipeline to assist determine these kind of points.

Incident 3: Untrained Mannequin Deployment

What was the difficulty?
Early on in our migration to deep studying fashions, we didn’t have many guardrails in place for mannequin deployments. For every mannequin variant we have been testing we’d retrain and mechanically redeploy the mannequin day by day, to be sure that the fashions have been skilled on latest information. Throughout one of many coaching runs, the mannequin coaching resulted in a mannequin that all the time predicted a 25% click on fee whatever the enter information and the ROC AUC metric on the validation information set was 0.5. We had basically deployed a mannequin to manufacturing that all the time predicted a 25% click on fee no matter any of the function inputs.

How was it discovered?
We first recognized the difficulty utilizing our system monitoring metrics in Datadog. We logged our click on predictions (p_ctr) as a histogram metric, and Datadog gives p50 and p99 aggregations. When the mannequin was deployed, we noticed the p50 and p99 values for the mannequin converge to the identical worth of ~25%, indicating that one thing had gone mistaken with the clicking prediction mannequin. We additionally reviewed the mannequin coaching logs and noticed that the metrics from the validation information set indicated a coaching error.

How was it fastened?
On this case, we have been in a position to rollback to the clicking mannequin from yesterday to resolve the difficulty, nevertheless it did take a while for the incident to be found and our rollback strategy on the time was considerably guide.

What did we study?
We discovered that this concern with dangerous mannequin coaching occurred round 2% of the time and wanted to arrange guardrails towards deploying these fashions. We added a mannequin validation module to our coaching pipeline that checked for thresholds on the validation metrics, and likewise in contrast the outputs of the brand new and prior runs on the mannequin on the identical information set. We additionally arrange alerts on Datadog to flag giant modifications within the p50 p_ctr metric and labored on automating our mannequin rollback course of.

Incident 4: Unhealthy Warmup Knowledge for TensorFlow Serving

What was the difficulty?
We used warmup information for TensorFlow serving to enhance the rollout time of recent mannequin deployments and to assist with serving latency. We bumped into a difficulty the place the tensors outlined within the warmup file didn’t correspond to the tensors outlined within the TensorFlow mannequin, leading to failed mannequin deployments.

How was it discovered?
In an early model of our in-house serving, this mismatch between warmup information and mannequin tensor definitions would trigger all mannequin serving to return to a halt and require a mannequin rollback to stabilize the system. That is one other incident that was initially captured by our operational metrics on Datadog, since we noticed a big spike in mannequin serving error requests. We confirmed that there was a difficulty with the newly deployed mannequin by deploying it to Vertex AI and confirming that the warmup information have been the basis explanation for the difficulty.

How was it fastened?
We up to date our mannequin deployment module to verify that the mannequin tensors and warmup information have been suitable by launching an area occasion of TensorFlow serving within the mannequin coaching pipeline and sending pattern requests utilizing the warmup file information. We additionally did extra guide testing with Vertex AI when launching new kinds of fashions with noticeably totally different tensor shapes.

What did we study?
We discovered that we wanted to have totally different environments for testing TensorFlow mannequin deployments earlier than pushing them to manufacturing. We have been in a position to do some testing with Vertex AI, however ultimately arrange a staging atmosphere for our in-house model of TensorFlow serving to offer a correct CI/CD atmosphere for mannequin deployment.

Incident 5: Problematic Time-Based mostly Options

What was the difficulty?
We explored some time-based options in our fashions, comparable to weeks_ago, to seize modifications in conduct over time. For the coaching pipeline, this function was calculated as flooring(date_diff(at this time, day_of_impression)/7). It was a extremely ranked function in a few of our fashions, nevertheless it additionally added unintended bias to our fashions. Throughout mannequin serving, this worth is all the time set to 0, since we’re making mannequin predictions in actual time, and at this time is similar as day_of_impression. The important thing concern is that the mannequin coaching pipeline was discovering patterns within the coaching information that will create bias points when making use of the mannequin on reside information.

How was it discovered?
This was one other incident that we discovered largely by instinct and later confirmed to be an issue by evaluating the implementation logic throughout the coaching and mannequin serving pipelines. We discovered that the mannequin serving pipeline all the time set the worth to 0 whereas the coaching pipeline used a variety of values on condition that we regularly use months previous examples for coaching.

How was it fastened?
We created a variant of the mannequin with all the relative time-based options eliminated and did an A/B check to check the efficiency of the variants. The mannequin that included the time based mostly options carried out higher on the holdout metrics throughout offline testing, however the mannequin with the options eliminated labored higher within the A/B check and we ended up eradicating the options from all the fashions.

What did we study?
We discovered that we had launched bias into our fashions in an unintended means. The options have been compelling to discover, as a result of person conduct does change over time, and introducing these options did lead to higher offline metrics for our fashions. Finally we determined to categorize these as problematic beneath the function parity class, the place we see variations in values between the mannequin coaching and serving pipelines.

Incident 6: Suggestions Options

What was the difficulty?
We had a function known as clearing_price that logged how excessive we have been keen to bid on an impression for a tool over the past time that we served an advert impression for the gadget. This was a helpful function, as a result of it helped us to bid on units with a excessive bid flooring, the place the mannequin wants excessive confidence {that a} conversion occasion will happen. This function by itself usually wasn’t problematic, nevertheless it did turn into an issue throughout an incident the place we launched dangerous labels into our coaching information set. We ran an experiment that resulted in false positives in our coaching information set, and we began to see a suggestions concern the place the mannequin bias grew to become a difficulty.

How was it discovered?
This was a really difficult incident to determine the basis explanation for, as a result of the experiment that generated the false constructive labels was run on a small cohort of visitors, so we didn’t see a sudden change in operational metrics like we did with a few of the different incidents in Datadog. As soon as we recognized which units and impressions have been impacted by this check, we regarded on the function drift of our information set and located that the common worth of the clearning_price function was rising steadily because the rollout of the experiment. The false positives within the label information have been the basis explanation for the incident, and the drift on this function was a secondary concern that was inflicting the fashions to make dangerous predictions.

How was it fastened?
Step one was to rollback to a best-known mannequin earlier than the problematic experiment was launched. We then cleaned up the information set and eliminated the false positives that we might determine from the coaching information set. We continued to see points and likewise made the decision to take away the problematic function from our fashions, much like the time-based options, to stop this function from creating future suggestions loops sooner or later.

What did we study?
We discovered that some options are useful for making the mannequin extra assured in predicting person conversions, however aren’t well worth the danger as a result of they’ll introduce a tail-spin impact the place the fashions rapidly deteriorate in efficiency and create incidents. To exchange the clearing value function, we launched new options utilizing the minimal bid to win values from public sale callbacks.

Incident 7: Unhealthy Function Encoding

What was the difficulty?
We explored just a few options that have been numeric and computed as ratios, comparable to the common click on fee of a tool, computed because the variety of clicks over the variety of impressions served to the gadget. We ran right into a function parity concern the place we dealt with divide by zero in numerous methods between the coaching and serving mannequin pipelines.

How was it discovered?
We’ve got a function parity verify the place we log the tensors created throughout mannequin inference for a subset of impressions and run the coaching pipeline on these impressions and evaluate the values generated within the coaching pipeline with the logged worth at serving time. We observed a big discrepancy for the ratio based mostly options and located that we encoded divide by zero as -1 within the coaching pipeline and 0 within the serving pipeline.

How was it fastened?
We up to date the serving pipeline to match the logic within the coaching pipeline, the place we set the worth to -1 when a divide by zero happens for the ratio based mostly options.

What did we study?
Our pipeline for detecting function parity points allowed us to rapidly determine the basis explanation for this concern as soon as the mannequin was deployed to manufacturing, nevertheless it’s additionally a state of affairs we wish to keep away from earlier than a mannequin is deployed. We utilized the identical studying from incident 2, the place we included information scientists on pull request critiques to assist determine potential points between our coaching and serving mannequin pipelines.

Incident 8: String Parsing

What was the difficulty?
We used a 1-hot encoding strategy the place we select the highest ok values, that are assigned indices from 1 to ok, and use 0 as an out-of-vocab (OOV) worth. We bumped into an issue with the encoding from strings to integers when coping with categorical options comparable to app bundle, which regularly has extra characters. For instance, the vocabulary could map the bundle com.dreamgames.royalmatch to index 3, however within the coaching pipeline the bundle is ready to com.dreamgames.royalmatch$hl=en_US and the worth will get encoded to 0, as a result of it’s thought of OOV. The core concern we bumped into was totally different logic for sanitizing string values between the coaching and serving pipelines earlier than making use of vocabularies.

How was it discovered?
This was one other incident that we found with our function parity checker. We discovered a number of examples the place one pipeline encoded the values as OOV whereas the opposite pipeline assigned non-zero values. We then in contrast the function values previous to encoding and observed discrepancies between how we did string parsing within the coaching and serving pipelines.

How was it fastened?
Our brief time period repair was to replace the coaching pipeline to carry out the identical string parsing logic because the serving pipeline. Long run we targeted on truncating the app bundle names on the information ingestion step, to scale back the necessity for guide parsing steps within the totally different pipelines.

What did we study?
We discovered that coping with problematic strings at information ingestion offered probably the most constant outcomes when coping with string values. We additionally bumped into points with unicode characters exhibiting up in app bundle names and labored to accurately parse these throughout ingestion. We additionally discovered it essential to often examine the vocabulary entries which are generated by the system to verify particular characters weren’t exhibiting up in entries.

Takeaways

Whereas it might be tempting to make use of deep studying in manufacturing for mannequin serving, there’s a whole lot of potential points you could encounter with reside mannequin serving. It’s vital to have sturdy plans in place for incident administration when working with machine studying fashions, in an effort to rapidly recuperate when mannequin efficiency turns into problematic and study from these missteps. On this submit we coated 8 totally different incidents I encountered when utilizing deep studying to foretell click on and set up conversion in a cellular AdTech platform. Right here’s are the important thing takeaways I discovered from these machine studying incidents:

• It’s vital to log function values, encoded values, tensor values, and mannequin predictions throughout mannequin serving, to make sure that you shouldn’t have function parity or mannequin parity points in your mannequin pipelines.
• Mannequin validation is a essential step in mannequin deployment and check environments might help cut back incidents.
• Watch out for the options that you just embody in your mannequin, they could be introducing bias or inflicting unintended suggestions.
• If in case you have totally different pipelines for mannequin coaching and mannequin serving, the group members engaged on the pipelines needs to be reviewing one another’s pull requests for ML function implementations.

Machine studying is a self-discipline that may study rather a lot from DevOps to scale back the prevalence of incidents, and MLOps ought to embody processes for effectively responding to points with ML fashions in manufacturing.