Now, we have now the options for our mannequin. We’ll cut up our knowledge into 3 units:
1- Coaching dataset : It’s the set of information the place we are going to practice our mannequin
2 – Check dataset : Knowledge used to judge the efficiency of our mannequin.
3- After modification dataset: Knowledge used to compute the uplift utilizing our mannequin.
from sklearn.model_selection import train_test_splitstart_modification_date = dt.datetime(2024, 2,1)
X_before_modification = X[X.index < start_modification_date]
y_before_modification = y[y.index < start_modification_date].kpi
X_after_modification = X[X.index >= start_modification_date]
y_after_modification = y[y.index >= start_modification_date].kpi
X_train, X_test , y_train , y_test = train_test_split(X_before_modification, y_before_modification, test_size= 0.25, shuffle = False)
Be aware : You need to use a fourth subset of information to carry out some mannequin choice. Right here we received’t do numerous mannequin choice, so it doesn’t matter so much. However it can should you begin to choose your mannequin amongst tenths of others.
Be aware 2: Cross-validation can also be very potential and beneficial.
Be aware 3 : I do advocate splitting knowledge with out shuffling (shuffling = False). It’ll enable you to concentrate on the eventual temporal drift of your mannequin.
from sklearn.ensemble import RandomForestRegressormannequin = RandomForestRegressor(min_samples_split=4)
mannequin.match(X_train, y_train)
y_pred = mannequin.predict(X_test)
And right here you practice your predictor. We use a random forest regressor for its comfort as a result of it permits us to deal with non-linearity, lacking knowledge, and outliers. Gradients Boosting Timber algorithms are additionally excellent for this use.
Many papers about Artificial Management will use linear regression right here, however we predict it isn’t helpful right here as a result of we’re not actually within the mannequin’s interpretability. Furthermore, decoding such regression might be difficult.
Counterfactual Analysis
Our prediction can be on the testing set. The primary speculation we are going to make is that the efficiency of the mannequin will keep the identical after we compute the uplift. That’s the reason we have a tendency to make use of numerous knowledge in our We contemplate 3 totally different key indicators to judge the standard of the counterfactual prediction :
1-Bias : Bias controls the presence of a niche between your counterfactual and the true knowledge. It’s a robust restrict in your capability to compute as a result of it received’t be decreased by ready extra time after the modification.
bias = float((y_pred - y_test).imply()/(y_before_modification.imply()))
bias
> 0.0030433481322823257
We typically specific the bias as a share of the common worth of the KPI. It’s smaller than 1%, so we must always not anticipate to measure results larger than that. In case your bias is just too large, you need to verify for a temporal drift (and add a development to your prediction). You can even right your prediction and deduce the bias from the prediction, supplied you management the impact of this correction of recent knowledge.
2-Commonplace Deviation σ: We additionally need to management how dispersed are the predictions across the true values. We due to this fact use the usual deviation, once more expressed as a share of the common worth of the kpi.
sigma = float((y_pred - y_test).std()/(y_before_modification.imply()))
sigma
> 0.0780972738325956
The excellent news is that the uncertainty created by the deviation is decreased when the variety of knowledge factors improve. We desire a predictor with out bias, so it may very well be needed to simply accept a rise within the deviation if allowed to restrict the bias.
It may also be attention-grabbing to take a look at bias and variance by trying on the distribution of the forecasting errors. It may be helpful to see if our calculation of bias and deviation is legitimate, or whether it is affected by outliers and excessive values.
import seaborn as sns
import matplotlib.pyplot as pltf, ax = plt.subplots(figsize=(8, 6))
sns.histplot(pd.DataFrame((y_pred - y_test)/y_past.imply()), x = 'kpi', bins = 35, kde = True, stat = 'likelihood')
f.suptitle('Relative Error Distribution')
ax.set_xlabel('Relative Error')
plt.present()
3- Auto-correlation α: Usually, errors are auto-correlated. It signifies that in case your prediction is above the true worth on a given day, it has extra probability of being above the following day. It’s a drawback as a result of most classical statistical instruments require independence between observations. What occurred on a given day ought to have an effect on the following one. We use auto-correlation as a measure of dependence between in the future and the following.
df_test = pd.DataFrame(zip(y_pred, y_test), columns = ['Prevision','Real'], index = y_test.index)
df_test = df_test.assign(
ecart = df_test.Prevision - df_test.Actual)
alpha = df_test.ecart.corr(df_test.ecart.shift(1))
alpha
> 0.24554635095548982
A excessive auto-correlation is problematic however might be managed. A potential causes for it are unobserved covariates. If as an example, the shop you need to measure organized a particular occasion, it might improve its gross sales for a number of days. It will result in an sudden sequence of days above the prevision.
df_test = pd.DataFrame(zip(y_pred, y_test), columns = ['Prevision','Reel'], index = y_test.index)f, ax = plt.subplots(figsize=(15, 6))
sns.lineplot(knowledge = df_test, x = 'date', y= 'Reel', label = 'True Worth')
sns.lineplot(knowledge = df_test, x = 'date', y= 'Prevision', label = 'Forecasted Worth')
ax.axvline(start_modification_date, ls = '--', shade = 'black', label = 'Begin of the modification')
ax.legend()
f.suptitle('KPI TX_1')
plt.present()
Within the determine above, you may see an illustration of the auto-correlation phenomenon. In late April 2023, for a number of days, forecasted values are above the true worth. Errors usually are not impartial of each other.
Impression Calculation
Now we are able to compute the affect of the modification. We evaluate the prediction after the modification with the precise worth. As all the time, it’s expressed as a share of the imply worth of the KPI.
y_pred_after_modification = mannequin.predict(X_after_modification)
uplift =float((y_after_modification - y_pred_after_modification).imply()/y_before_modification.imply())
uplift
> 0.04961773643584396
We get a relative improve of 4.9% The “true” worth (the info used have been artificially modified) was 3.0%, so we’re not removed from it. And certainly, the true worth is usually above the prediction :
We are able to compute a confidence interval for this worth. If our predictor has no bias, the scale of its confidence interval might be expressed with:
The place σ is the usual deviation of the prediction, α its auto-correlation, and N the variety of days after the modification.
N = y_after_modification.form[0]
ec = sigma/(sqrt(N) *(1-alpha))print('68%% IC : [%.2f %% , %.2f %%]' % (100*(uplift - ec),100 * (uplift + ec) ))
print('95%% IC : [%.2f %% , %.2f %%]' % (100*(uplift -2 *ec),100 * (uplift +2*ec) ))
68% IC : [3.83 % , 6.09 %]
95% IC : [2.70 % , 7.22 %]
The vary of the 95% CI is round 4.5% for 84 days. It’s affordable for a lot of purposes, as a result of it’s potential to run an experiment or a proof of idea for 3 months.
Be aware: the boldness interval could be very delicate to the deviation of the preliminary predictor. That’s the reason it’s a good suggestion to take a while to carry out mannequin choice (on the coaching set solely) earlier than deciding on a superb mannequin.
Mathematical formulation of the mannequin
To date we have now tried to keep away from maths, to permit for a neater comprehension. On this part, we are going to current the mathematical mannequin beneath the mannequin.