Calculating the Uncertainty Coefficient (Theil’s U) in Python | by Marc Linder | Oct, 2024

Introduction

Theil’s U, often known as the uncertainty coefficient or entropy coefficient, quantifies the power of affiliation between two nominal variables. It assesses how a lot figuring out the worth of 1 variable reduces uncertainty concerning the different, offering a measure of affiliation that ranges from 0 to 1. A better worth signifies a stronger relationship, making Thiel’s U notably helpful in fields corresponding to statistics and knowledge science for exploring relationships inside categorical knowledge.

Concept

Theil’s U is a measure of nominal affiliation primarily based on the idea of data entropy. Suppose we now have samples from two discrete random variables, X and Y.

Then the entropy of X is outlined as:

And the conditional entropy of X given Y is outlined as:

We are able to then use the joint distribution (numerator) together with the marginal chances of X or Y to calculate the conditional distributions of X given Y (denominator) or Y given X, respectively, as follows:

The outcome captures how the chance of 1 variable modifications given the worth of the opposite. We are able to calculate the chance of X given Y by utilizing the joint chance of X and Y — that’s, the chance of various combos of X and Y — in addition to the marginal chance of Y. We insert the results of their division into our components for H(X) to acquire:

A lot for the idea; right here’s how we are able to calculate the conditional entropy of X given Y in Python.

from typing import Checklist, Union
from collections import Counter
import math
def conditional_entropy(
x: Checklist[Union[int, float]], 
y: Checklist[Union[int, float]]
) -> float:
""" Calculates conditional entropy """
# Depend distinctive values
y_counter = Counter(y)  # Counts of distinctive values in y
xy_counter = Counter(listing(zip(x, y)))  # Counts of distinctive pairs from (x, y)
# Calculate sum of y values
total_occurrences = sum(y_counter.values())
# (Re-)set entropy to 0
entropy = 0
# For each distinctive worth pair of x and y
for xy in xy_counter.keys():
# Joint chance of x AND y
p_xy = xy_counter[xy] / total_occurrences
# Marginal chance of y
p_y = y_counter[xy[1]] / total_occurrences
# Conditional chance of x given y
p_x_given_y = p_xy / p_y
# Calculate the conditional entropy H(X|Y)
entropy += p_xy * math.log(p_x_given_y, 2)  # Use base 2 as a substitute of pure (base e)
return -entropy

As soon as we now have calculated the conditional entropy of X given Y, we are able to calculate Theil’s U. One final step is to calculate the entropy of X, which we outlined initially of this text. The uncertainty coefficient, or proficiency, is then calculated as follows:

Switching from concept to observe, this may be achieved in Python utilizing the next code:

import scipy.stats as ss
def theils_u(
x: Checklist[Union[int, float]], 
y: Checklist[Union[int, float]]
) -> float:
""" Calculate Theil U """
# Calculate conditional entropy of x and y
H_xy = conditional_entropy(x,y)
# Depend distinctive values
x_counter = Counter(x)
# Calculate sum of x values
total_occurrences = sum(x_counter.values())
# Convert all absolute counts of x values in x_counter to chances
p_x = listing(map(lambda depend: depend/total_occurrences, x_counter.values()))
# Calculate entropy of single distribution x
H_x = ss.entropy(p_x)
return (H_x - H_xy) / H_x if H_x != 0 else 0

Lastly we are able to then outline a operate that calculates the Theil’s values for each characteristic mixture inside a given dataset. We are able to do that in Python with the next code:

import itertools
import pandas as pd
def get_theils_u_for_df(df: pd.DataFrame) -> pd.DataFrame:
""" Compute Theil's U for each characteristic mixture within the enter df """
# Create an empty dataframe to fill
theilu = pd.DataFrame(index=df.columns, columns=df.columns)
# Insert Theil U values into empty dataframe
for var1, var2 in itertools.combos(df, 2):
u = theil_u(df[var1],df[var2])
theilu[var1][var2] = spherical(u, 2) # fill decrease diagonal
u = theil_u(df[var2],df[var1])   
theilu[var2][var1] = spherical(u, 2) # fill higher diagonal
# Set 1s to diagonal the place row index + column index == n - 1
for i in vary(0, len(theilu.columns)):
for j in vary(0, len(theilu.columns)):
if i == j:
theilu.iloc[i, j] = 1
# Convert all values within the DataFrame to drift
return theilu.map(float)

Code Instance

We’ll display the performance of the code utilizing the well-known Iris dataset. Along with its numeric variables, the dataset accommodates a categorical variable, “species.” Conventional correlation measures, corresponding to Pearson’s correlation, are restricted in capturing relationships between categorical and numerical options. Nevertheless, Thiel’s U can successfully measure the affiliation between “species” and the opposite numerical options.

import pandas as pd
import seaborn as sns
import itertools
import matplotlib.pyplot as plt
# Load the Iris dataset from seaborn
df = sns.load_dataset('iris')
# Compute Theil's U for each characteristic mixture within the enter df
theilu = get_theils_u_for_df(df)
# Create a heatmap of the Theil's V values
plt.determine(figsize=(10, 4))
sns.heatmap(theilu, annot=True, cmap='Reds', fmt='.2f')
plt.title('Heatmap of Theil's U for all variable pairs') 
plt.present()

The result’s a heatmap of Thiel’s U for all variable pairs. Be aware that this measure has the benefit of being uneven, which means the connection between two variables can differ relying on the path of research. For instance, Thiel’s U can quantify how a lot info X gives about Y, which might not be the identical as how a lot info Y gives about X.

The interpretation of the outcomes is comparatively simple: Petal Size and Petal Width have the strongest associations with the explicit variable “species,” each with a worth of 0.91. This means that figuring out the petal dimensions gives a excessive diploma of details about the flower species. Sepal Size additionally has a reasonable relationship with species at 0.55, which means it affords some details about the species, although lower than the petal measurements. Sepal Width has the weakest affiliation with species at 0.33, indicating it gives comparatively little details about the flower sort. The comparatively decrease values between the sepal measurements and species spotlight that petal dimensions are extra informative for predicting species, which is per the recognized traits of the Iris dataset.

Conclusion

On this article, we demonstrated learn how to calculate Theil’s U to evaluate associations between categorical and numerical variables. By making use of this measure to the Iris dataset, we confirmed that petal dimensions present important insights into predicting flower species, highlighting the effectiveness of Theil’s U in comparison with conventional correlation strategies.

Sources

• Theil, H. (1958): Financial Forecasts and Coverage. Amsterdam: North Holland.
• Theil, H. (1966): Utilized Financial Forecasting. Chicago: Rand McNally.
• Bliemel, F. (1973): Theil’s Forecast Accuracy Coefficient: A Clarification, Journal of Advertising and marketing Analysis 10(4), pp. 444–446

Be aware: Until in any other case famous, all photos are by the creator.