Detecting Anomalies in Social Media Quantity Time Sequence | by Lorenzo Mezzini | Nov, 2024

Analyzing a Pattern Twitter Quantity Dataset

Let’s begin by loading and visualizing a pattern Twitter quantity dataset for Apple:

From this plot, we will see that there are a number of spikes (anomalies) in our knowledge. These spikes in volumes are those we wish to establish.

Trying on the second plot (log-scale) we will see that the Twitter quantity knowledge reveals a transparent every day cycle, with larger exercise through the day and decrease exercise at evening. This seasonal sample is widespread in social media knowledge, because it displays the day-night exercise of customers. It additionally presents a weekly seasonality, however we’ll ignore it.

Eradicating Seasonal Developments

We wish to be sure that this cycle doesn’t intervene with our conclusions, thus we’ll take away it. To take away this seasonality, we’ll carry out a seasonal decomposition.

First, we’ll calculate the shifting common (MA) of the quantity, which can seize the pattern. Then, we’ll compute the ratio of the noticed quantity to the MA, which provides us the multiplicative seasonal impact.

As anticipated, the seasonal pattern follows a day/evening cycle with its peak through the day hours and its saddle at nighttime.

To additional proceed with the decomposition we have to calculate the anticipated worth of the quantity given the multiplicative pattern discovered earlier than.

Analyzing Residuals and Detecting Anomalies

The ultimate element of the decomposition is the error ensuing from the subtraction between the anticipated worth and the true worth. We are able to contemplate this measure because the de-meaned quantity accounting for seasonality:

Curiously, the residual distribution carefully follows a Pareto distribution. This property permits us to make use of the Pareto distribution to set a threshold for detecting anomalies, as we will flag any residuals that fall above a sure percentile (e.g., 0.9995) as potential anomalies.

Now, I’ve to do a giant disclaimer: this property I’m speaking about will not be “True” per se. In my expertise in social listening, I’ve noticed that holds true with most social knowledge. Aside from some proper skewness in a dataset with many anomalies.

On this particular case, we’ve properly over 15k observations, therefore we’ll set the p-value at 0.9995. Given this threshold, roughly 5 anomalies for each 10.000 observations will likely be detected (assuming an ideal Pareto distribution).

Subsequently, if we examine which statement in our knowledge has an error whose p-value is larger than 0.9995, we get the next indicators:

From this graph, we see that the observations with the very best volumes are highlighted as anomalies. In fact, if we need extra or fewer indicators, we will alter the chosen p-value, retaining in thoughts that, because it decreases, it can improve the variety of indicators.