Information scientists are within the enterprise of decision-making. Our work is targeted on easy methods to make knowledgeable selections beneath uncertainty.
And but, in relation to quantifying that uncertainty, we regularly lean on the thought of “statistical significance” — a device that, at greatest, supplies a shallow understanding.
On this article, we’ll discover why “statistical significance” is flawed: arbitrary thresholds, a false sense of certainty, and a failure to handle real-world trade-offs.
Most essential, we’ll learn to transfer past the binary mindset of great vs. non-significant, and undertake a decision-making framework grounded in financial affect and threat administration.
Think about we simply ran an A/B take a look at to guage a brand new characteristic designed to spice up the time customers spend on our web site — and, because of this, their spending.
The management group consisted of 5,000 customers, and the therapy group included one other 5,000 customers. This provides us two arrays, named therapy
and management
, every of them containing 5,000 values representing the spending of particular person customers of their respective teams.