Throughout the telecommunication growth, Claude Shannon, in his seminal 1948 paper¹, posed a query that will revolutionise expertise:
How can we quantify communication?
Shannon’s findings stay basic to expressing info quantification, storage, and communication. These insights made main contributions to the creation of applied sciences starting from sign processing, information compression (e.g., Zip information and compact discs) to the Web and synthetic intelligence. Extra broadly, his work has considerably impacted various fields corresponding to neurobiology, statistical physics and laptop science (e.g, cybersecurity, cloud computing, and machine studying).
[Shannon’s paper is the]
Magna Carta of the Data Age
That is the primary article in a sequence that explores info quantification – a vital software for information scientists. Its purposes vary from enhancing statistical analyses to serving as a go-to determination heuristic in cutting-edge machine studying algorithms.
Broadly talking, quantifying info is assessing uncertainty, which can be phrased as: “how stunning is an end result?”.
This text thought shortly grew right into a sequence since I discovered this subject each fascinating and various. Most researchers, at one stage or one other, come throughout generally used metrics corresponding to entropy, cross-entropy/KL-divergence and mutual-information. Diving into this subject I discovered that as a way to absolutely admire these one must be taught a bit concerning the fundamentals which we cowl on this first article.
By studying this sequence you’ll achieve an instinct and instruments to quantify:
- Bits/Nats – Unit measures of knowledge.
- Self-Data – **** The quantity of knowledge in a selected occasion.
- Pointwise Mutual Data – The quantity of knowledge shared between two particular occasions.
- Entropy – The typical quantity of knowledge of a variable’s end result.
- Cross-entropy – The misalignment between two chance distributions (additionally expressed by its by-product KL-Divergence – a distance measure).
- Mutual Data – The co-dependency of two variables by their conditional chance distributions. It expresses the knowledge achieve of 1 variable given one other.
No prior information is required – only a fundamental understanding of chances.
I display utilizing frequent statistics corresponding to coin and cube 🎲 tosses in addition to machine studying purposes corresponding to in supervised classification, characteristic choice, mannequin monitoring and clustering evaluation. As for actual world purposes I’ll focus on a case examine of quantifying DNA range 🧬. Lastly, for enjoyable, I additionally apply to the favored mind tornado generally referred to as the Monty Corridor downside 🚪🚪 🐐 .
All through I present python code 🐍 , and attempt to preserve formulation as intuitive as potential. In case you have entry to an built-in growth setting (IDE) 🖥 you would possibly need to plug 🔌 and play 🕹 round with the numbers to realize a greater instinct.
This sequence is split into 4 articles, every exploring a key facet of Data Idea:
-
😲 Quantifying Shock: 👈 👈 👈 YOU ARE HERE
On this opening article, you’ll learn to quantify the “shock” of an occasion utilizing _self-informatio_n and perceive its items of measurement, corresponding to _bit_s and _nat_s. Mastering self-information is crucial for constructing instinct concerning the subsequent ideas, as all later heuristics are derived from it. - 🤷 Quantifying Uncertainty: Constructing on self-information, this text shifts focus to the uncertainty – or “common shock” – related to a variable, referred to as entropy. We’ll dive into entropy’s wide-ranging purposes, from Machine Studying and information evaluation to fixing enjoyable puzzles, showcasing its adaptability.
- 📏 Quantifying Misalignment: Right here, we’ll discover the right way to measure the gap between two chance distributions utilizing entropy-based metrics like cross-entropy and KL-divergence. These measures are significantly precious for duties like evaluating predicted versus true distributions, as in classification loss capabilities and different alignment-critical eventualities.
- 💸 Quantifying Acquire: Increasing from single-variable measures, this text investigates the relationships between two. You’ll uncover the right way to quantify the knowledge gained about one variable (e.g, goal Y) by realizing one other (e.g., predictor X). Purposes embody assessing variable associations, characteristic choice, and evaluating clustering efficiency.
Every article is crafted to face alone whereas providing cross-references for deeper exploration. Collectively, they supply a sensible, data-driven introduction to info concept, tailor-made for information scientists, analysts and machine studying practitioners.
Disclaimer: Until in any other case talked about the formulation analysed are for categorical variables with c≥2 courses (2 that means binary). Steady variables will probably be addressed in a separate article.
🚧 Articles (3) and (4) are presently beneath building. I’ll share hyperlinks as soon as obtainable. Comply with me to be notified 🚧
Quantifying Shock with Self-Data
Self-information is taken into account the constructing block of knowledge quantification.
It’s a method of quantifying the quantity of “shock” of a selected end result.
Formally self-information, or additionally known as Shannon Data or info content material, quantifies the shock of an occasion x occurring based mostly on its chance, p(x). Right here we denote it as hₓ:
The items of measure are known as bits. One bit (binary digit) is the quantity of knowledge for an occasion x that has chance of p(x)=½. Let’s plug in to confirm: hₓ=-log₂(½)= log₂(2)=1 bit.
This heuristic serves as an alternative choice to chances, odds and log-odds, with sure mathematical properties that are advantageous for info concept. We focus on these under when studying about Shannon’s axioms behind this selection.
It’s at all times informative to discover how an equation behaves with a graph:
To deepen our understanding of self-information, we’ll use this graph to discover the mentioned axioms that justify its logarithmic formulation. Alongside the way in which, we’ll additionally construct instinct about key options of this heuristic.
To stress the logarithmic nature of self-information, I’ve highlighted three factors of curiosity on the graph:
- At p=1 an occasion is assured, yielding no shock and therefore zero bits of knowledge (zero bits). A helpful analogy is a trick coin (the place each side present HEAD).
- Lowering the chance by an element of two (p=½) will increase the knowledge to _hₓ=_1 bit. This, after all, is the case of a good coin.
- Additional decreasing it by an element of 4 leads to hₓ(p=⅛)=3 bits.
In case you are fascinated about coding the graph here’s a python script:
To summarise this part:
Self-Data hₓ=-log₂(p(x)) quantifies the quantity of “shock” of a selected end result x.
Three Axioms
Referencing prior work by Ralph Hartley, Shannon selected -log₂(p) as a fashion to fulfill three axioms. We’ll use the equation and graph to look at how these are manifested:
-
An occasion with chance 100% is no surprise and therefore doesn’t yield any info.
Within the trick coin case that is evident by p(x)=1 yielding hₓ=0. -
Much less possible occasions are extra stunning and supply extra info.
That is obvious by self-information reducing monotonically with rising chance. - The property of Additivity – the whole self-information of two unbiased occasions equals the sum of particular person contributions. This will probably be explored additional within the upcoming fourth article on Mutual Data.
There are mathematical proofs (that are past the scope of this sequence) that present that solely the log perform adheres to all three².
The appliance of those axioms reveals a number of intriguing and sensible properties of self-information:
Vital properties :
- Minimal certain: The primary axiom hₓ(p=1)=0 establishes that self-information is non-negative, with zero as its decrease certain. That is extremely sensible for a lot of purposes.
- Monotonically reducing: The second axiom ensures that self-information decreases monotonically with rising chance.
- No Most certain: On the excessive the place _p→_0, monotonicity results in self-information rising with out certain hₓ(_p→0) →_ ∞, a characteristic that requires cautious consideration in some contexts. Nevertheless, when averaging self-information – as we’ll later see within the calculation of entropy – chances act as weights, successfully limiting the contribution of extremely unbelievable occasions to the general common. This relationship will turn into clearer after we discover entropy intimately.
It’s helpful to grasp the shut relationship to log-odds. To take action we outline p(x) because the chance of occasion x to occur and p(¬x)=1-p(x) of it to not occur. log-odds(x) = log₂(p(x)/p(¬x))= h(¬x) – h(x).
The primary takeaways from this part are
Axiom 1: An occasion with chance 100% is no surprise
Axiom 2: Much less possible occasions are extra stunning and, once they happen, present extra info.
Self info (1) monotonically decreases (2) with a minimal certain of zero and (3) no higher certain.
Within the subsequent two sections we additional focus on items of measure and selection of normalisation.
Data Models of Measure
Bits or Shannons?
A bit, as talked about, represents the quantity of knowledge related to an occasion that has a 50% chance of occurring.
The time period can be typically known as a Shannon, a naming conference proposed by mathematician and physicist David MacKay to keep away from confusion with the time period ‘bit’ within the context of digital processing and storage.
After some deliberation, I made a decision to make use of ‘bit’ all through this sequence for a number of causes:
- This sequence focuses on quantifying info, not on digital processing or storage, so ambiguity is minimal.
- Shannon himself, inspired by mathematician and statistician John Tukey, used the time period ‘bit’ in his landmark paper.
- ‘Bit’ is the usual time period in a lot of the literature on info concept.
- For comfort – it’s extra concise
Normalisation: Log Base 2 vs. Pure
All through this sequence we use base 2 for logarithms, reflecting the intuitive notion of a 50% probability of an occasion as a basic unit of knowledge.
Another generally utilized in machine studying is the pure logarithm, which introduces a distinct unit of measure known as nats (quick for natural items of knowledge). One nat corresponds to the knowledge gained from an occasion occurring with a chance of 1/e the place e is Euler’s quantity (≈2.71828). In different phrases, 1 nat = -ln(p=(1/e)).
The connection between bits (base 2) and nats (pure log) is as follows:
1 bit = ln(2) nats ≈ 0.693 nats.
Consider it as just like a financial present trade or changing centimeters to inches.
In his seminal publication Shanon defined that the optimum selection of base depends upon the particular system being analysed (paraphrased barely from his authentic work):
- “A tool with two secure positions […] can retailer one bit of knowledge” (bit as in binary digit).
- “A digit wheel on a desk computing machine that has ten secure positions […] has a storage capability of 1 decimal digit.”³
- “In analytical work the place integration and differentiation are concerned the bottom e is typically helpful. The ensuing items of knowledge will probably be known as pure items.“
Key points of machine studying, corresponding to in style loss capabilities, typically depend on integrals and derivatives. The pure logarithm is a sensible selection in these contexts as a result of it may be derived and built-in with out introducing extra constants. This possible explains why the machine studying group ceaselessly makes use of nats because the unit of knowledge – it simplifies the arithmetic by avoiding the necessity to account for elements like ln(2).
As proven earlier, I personally discover base 2 extra intuitive for interpretation. In instances the place normalisation to a different base is extra handy, I’ll make an effort to clarify the reasoning behind the selection.
To summarise this part of items of measure:
bit = quantity of knowledge to differentiate between two equally possible outcomes.
Now that we’re acquainted with self-information and its unit of measure let’s study a couple of use instances.
Quantifying Occasion Data with Cash and Cube
On this part, we’ll discover examples to assist internalise the self-information axioms and key options demonstrated within the graph. Gaining a strong understanding of self-information is crucial for greedy its derivatives, corresponding to entropy, cross-entropy (or KL divergence), and mutual info – all of that are averages over self-information.
The examples are designed to be easy, approachable, and lighthearted, accompanied by sensible Python code that will help you experiment and construct instinct.
Word: When you really feel comfy with self-information, be happy to skip these examples and go straight to the Quantifying Uncertainty article.
To additional discover the self-information and bits, I discover analogies like coin flips and cube rolls significantly efficient, as they’re typically helpful analogies for real-world phenomena. Formally, these could be described as multinomial trials with n=1 trial. Particularly:
- A coin flip is a Bernoulli trial, the place there are c=2 potential outcomes (e.g., heads or tails).
- Rolling a die represents a categorical trial, the place c≥3 outcomes are potential (e.g., rolling a six-sided or eight-sided die).
As a use case we’ll use simplistic climate stories restricted to that includes solar 🌞 , rain 🌧 , and snow ⛄️.
Now, let’s flip some digital cash 👍 and roll some funky-looking cube 🎲 …
Truthful Cash and Cube
We’ll begin with the only case of a good coin (i.e, 50% probability for achievement/Heads or failure/Tails).
Think about an space for which at any given day there’s a 50:50 probability for solar or rain. We are able to write the chance of every occasion be: p(🌞 )=p(🌧 )=½.
As seen above, in accordance the the self-information formulation, when 🌞 or 🌧 is reported we’re offered with h(🌞 __ )=h(🌧 )=-log₂(½)=1 bit of knowledge.
We are going to proceed to construct on this analogy afterward, however for now let’s flip to a variable that has greater than two outcomes (c≥3).
Earlier than we tackle the usual six sided die, to simplify the maths and instinct, let’s assume an 8 sided one (_c=_8) as in Dungeons Dragons and different tabletop video games. On this case every occasion (i.e, touchdown on both sides) has a chance of p(🔲 ) = ⅛.
When a die lands on one aspect going through up, e.g, worth 7️⃣, we’re supplied with h(🔲 =7️⃣)=-log₂(⅛)=3 bits of knowledge.
For the standard six sided truthful die: p(🔲 ) = ⅙ → an occasion yields __ h(🔲 )=-log₂(⅙)=2.58 bits.
Evaluating the quantity of knowledge from the truthful coin (1 bit), 6 sided die (2.58 bits) and eight sided (3 bits) we determine the second axiom: The much less possible an occasion is, the extra stunning it’s and the extra info it yields.
Self info turns into much more fascinating when chances are skewed to favor sure occasions.
Loaded Cash and Cube
Let’s assume a area the place p(🌞 ) = ¾ and p(🌧 )= ¼.
When rain is reported the quantity of knowledge conveyed isn’t 1 bit however moderately h(🌧 )=-log₂(¼)=2 bits.
When solar is reported much less info is conveyed: h(🌞 )=-log₂(¾)=0.41 bits.
As per the second axiom— a rarer occasion, like p(🌧 )=¼, reveals extra info than a extra possible one, like p(🌞 )=¾ – and vice versa.
To additional drive this level let’s now assume a desert area the place p(🌞 ) =99% and p(🌧 )= 1%.
If sunshine is reported – that’s form of anticipated – so nothing a lot is learnt (“nothing new beneath the solar” 🥁) and that is quantified as h(🌞 )=0.01 bits. If rain is reported, nonetheless, you’ll be able to think about being fairly shocked. That is quantified as h(🌧 )=6.64 bits.
Within the following python scripts you’ll be able to study all of the above examples, and I encourage you to play with your personal to get a sense.
First let’s outline the calculation and printout perform:
import numpy as np
def print_events_self_information(probs):
for ps in probs:
print(f"Given distribution {ps}")
for occasion in ps:
if ps[event] != 0:
self_information = -np.log2(ps[event]) #similar as: -np.log(ps[event])/np.log(2)
text_ = f'When `{occasion}` happens {self_information:0.2f} bits of knowledge is communicated'
print(text_)
else:
print(f'a `{occasion}` occasion can't occur p=0 ')
print("=" * 20)Subsequent we’ll set a couple of instance distributions of climate frequencies
# Setting a number of chance distributions (every sums to 100%)
# Enjoyable reality - 🐍 💚 Emojis!
probs = [{'🌞 ': 0.5, '🌧 ': 0.5}, # half-half
{'🌞 ': 0.75, '🌧 ': 0.25}, # more sun than rain
{'🌞 ': 0.99, '🌧 ': 0.01} , # mostly sunshine
]
print_events_self_information(probs)This yields printout
Given distribution {'🌞 ': 0.5, '🌧 ': 0.5}
When `🌞 ` happens 1.00 bits of knowledge is communicated
When `🌧 ` happens 1.00 bits of knowledge is communicated
====================
Given distribution {'🌞 ': 0.75, '🌧 ': 0.25}
When `🌞 ` happens 0.42 bits of knowledge is communicated
When `🌧 ` happens 2.00 bits of knowledge is communicated
====================
Given distribution {'🌞 ': 0.99, '🌧 ': 0.01}
When `🌞 ` happens 0.01 bits of knowledge is communicated
When `🌧 ` happens 6.64 bits of knowledge is communicated Let’s study a case of a loaded three sided die. E.g, info of a climate in an space that stories solar, rain and snow at uneven chances: p(🌞 ) = 0.2, p(🌧 )=0.7, p(⛄️)=0.1.
Operating the next
print_events_self_information([{'🌞 ': 0.2, '🌧 ': 0.7, '⛄️': 0.1}])yields
Given distribution {'🌞 ': 0.2, '🌧 ': 0.7, '⛄️': 0.1}
When `🌞 ` happens 2.32 bits of knowledge is communicated
When `🌧 ` happens 0.51 bits of knowledge is communicated
When `⛄️` happens 3.32 bits of knowledge is communicated What we noticed for the binary case applies to increased dimensions.
To summarise – we clearly see the implications of the second axiom:
- When a extremely anticipated occasion happens – we don’t be taught a lot, the bit rely is low.
- When an surprising occasion happens – we be taught so much, the bit rely is excessive.
Occasion Data Abstract
On this article we launched into a journey into the foundational ideas of knowledge concept, defining the right way to measure the shock of an occasion. Notions launched function the bedrock of many instruments in info concept, from assessing information distributions to unraveling the inside workings of machine studying algorithms.
By means of easy but insightful examples like coin flips and cube rolls, we explored how self-information quantifies the unpredictability of particular outcomes. Expressed in bits, this measure encapsulates Shannon’s second axiom: rarer occasions convey extra info.
Whereas we’ve targeted on the knowledge content material of particular occasions, this naturally results in a broader query: what’s the common quantity of knowledge related to all potential outcomes of a variable?
Within the subsequent article, Quantifying Uncertainty, we construct on the inspiration of self-information and bits to discover entropy – the measure of common uncertainty. Removed from being only a stunning theoretical assemble, it has sensible purposes in information evaluation and machine studying, powering duties like determination tree optimisation, estimating range and extra.
Cherished this submit? ❤️🍕
💌 Comply with me right here, be a part of me on LinkedIn or 🍕 purchase me a pizza slice!
About This Sequence
Though I’ve twenty years of expertise in information evaluation and predictive modelling I at all times felt fairly uneasy about utilizing ideas in info concept with out actually understanding them.
The aim of this sequence was to place me extra relaxed with ideas of knowledge concept and hopefully present for others the reasons I wanted.
Take a look at my different articles which I wrote to higher perceive Causality and Bayesian Statistics:
Footnotes
¹ A Mathematical Idea of Communication, Claude E. Shannon, Bell System Technical Journal 1948.
It was later renamed to a e book The Mathematical Idea of Communication in 1949.
[Shannon’s “A Mathematical Theory of Communication”] the blueprint for the digital period – Historian James Gleick
² See Wikipedia web page on Data Content material (i.e, self-information) for an in depth derivation that solely the log perform meets all three axioms.
³ The decimal-digit was later renamed to a hartley (image Hart), a ban or a dit. See Hartley (unit) Wikipedia web page.
Credit
Until in any other case famous, all photographs have been created by the creator.
Many because of Will Reynolds and Pascal Bugnion for his or her helpful feedback.