Why “Statistical Significance” Is Pointless | by Samuele Mazzanti | Dec, 2024

Right here’s a greater framework for data-driven decision-making

[Image by Author]

Information scientists are within the enterprise of decision-making. Our work is targeted on easy methods to make knowledgeable selections beneath uncertainty.

And but, in relation to quantifying that uncertainty, we regularly lean on the thought of “statistical significance” — a device that, at greatest, supplies a shallow understanding.

On this article, we’ll discover why “statistical significance” is flawed: arbitrary thresholds, a false sense of certainty, and a failure to handle real-world trade-offs.

Most essential, we’ll learn to transfer past the binary mindset of great vs. non-significant, and undertake a decision-making framework grounded in financial affect and threat administration.

Think about we simply ran an A/B take a look at to guage a brand new characteristic designed to spice up the time customers spend on our web site — and, because of this, their spending.

The management group consisted of 5,000 customers, and the therapy group included one other 5,000 customers. This provides us two arrays, named therapy and management, every of them containing 5,000 values representing the spending of particular person customers of their respective teams.