Heatmaps for Time Collection  | In direction of Information Science

In 2015, the Wall Avenue Journal (WSJ) printed a extremely efficient sequence of heatmaps illustrating the impression of vaccines on infectious ailments in the USA. These visualizations showcased the ability of blanket insurance policies to drive widespread change. You may view the heatmaps right here.

Heatmaps are a flexible software for knowledge evaluation. Their means to facilitate comparative evaluation, spotlight temporal traits, and allow sample recognition makes them invaluable for speaking advanced info. 

On this Fast Success Information Science mission, we’ll use Python’s Matplotlib graphing library to recreate the WSJ’s measles chart, demonstrating tips on how to leverage heatmaps and punctiliously designed colorbars to affect knowledge storytelling.

The information 

The illness knowledge comes from the College of Pittsburgh’s Undertaking Tycho. This group works with nationwide and world well being institutes and researchers to make knowledge simpler to make use of to enhance world well being. The measles knowledge is on the market underneath a Artistic Commons Attribution 4.0 Worldwide Public License. 

For comfort, I’ve downloaded the info from Undertaking Tycho’s knowledge portal to a CSV file and saved it on this Gist. Later, we’ll entry it programmatically via the code.

The measles heatmap

We’ll use the Matplotlib pcolormesh() operate to assemble an in depth facsimile of the WSJ measles heatmap. Whereas different libraries, comparable to Seaborn, Plotly Specific, and hvplot, embody devoted heatmap features, these are constructed for ease of use, with a lot of the design selections abstracted away. This makes it troublesome to drive their outcomes to match the WSJ heatmap. 

Apart from pcolormesh(), Matplotlib’s imshow() operate (for “picture present”) may produce heatmaps. The pcolormesh operate, nevertheless, higher aligns gridlines with cell edges. 

Right here’s an instance of a heatmap made with imshow() that you simply examine to the pcolormesh() outcomes later. The primary distinction is the dearth of gridlines.

In 1963, the measles vaccine was licensed and launched throughout America with widespread uptake. Inside 5 years, the incidence of the illness was drastically decreased. By 2000, measles had been thought of eradicated in the USA, with any new circumstances arriving from outdoors the nation. Discover how nicely the visualization conveys this “huge image” whereas preserving the state-level particulars. That is due in no small half to the selection of colorbar.

The colours used within the visualization are biased. Greater than 80% of the colorbar consists of heat colours, and (mild) blue is reserved for the smallest values. This makes it simple to demarcate the pre- and post-vaccination durations. White cells denote lacking knowledge, represented by NaN (Not a Quantity) values. 

Examine the earlier heatmap to 1 constructed with a extra balanced colorbar:

The darker blue shade not solely overpowers the plot, it’s exhausting on the eyes. And whereas it’s nonetheless attainable to see the impact of the vaccine, the visible impression is way extra delicate than within the plot with the biased colorbar. Alternately, it’s simpler to parse larger values however on the expense of the general theme.

The code

The next code was written in JupyterLab and is offered by cell.

Importing libraries

The primary cell imports the libraries we’ll want to finish the mission. An internet seek for the library names will lead you to the set up directions.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colours import LinearSegmentedColormap, Normalize
from matplotlib.cm import ScalarMappable
import pandas as pd

Creating the customized colormap

The next code intently reproduces the colormap utilized by the WSJ. I used the net Picture Shade Picker software to establish the important thing colours from a screenshot of their measles heatmap and adjusted these based mostly on colours chosen for a related tutorial constructed for R.

# Normalize RGB colours:
colours = ['#e7f0fa',  # lightest blue
          '#c9e2f6',  # light blue
          '#95cbee',  # blue
          '#0099dc',  # dark blue
          '#4ab04a',  # green
          '#ffd73e',  # yellow
          '#eec73a',  # yellow brown
          '#e29421',  # dark tan
          '#f05336',  # orange
          '#ce472e']  # purple

# Create a listing of positions for every shade within the colormap:
positions = [0, 0.02, 0.03, 0.09, 0.1, 0.15, 0.25, 0.4, 0.5, 1]

# Create a LinearSegmentedColormap (steady colours):
custom_cmap = LinearSegmentedColormap.from_list('custom_colormap', 
                                                listing(zip(positions, 
                                                         colours)))

# Show a colorbar with the customized colormap:
fig, ax = plt.subplots(figsize=(6, 1))

plt.imshow([list(range(256))],
           cmap=custom_cmap, 
           facet='auto', 
           vmin=0, vmax=255)

plt.xticks([]), plt.yticks([])
plt.present()

Right here’s the generic colorbar produced by the code:

This code makes a steady colormap utilizing Matplotlib’s LinearSegmentedColormap() class. This class specifies colormaps utilizing anchor factors between which RGB(A) values are interpolated. That’s, it generates colormap objects based mostly on lookup tables utilizing linear segments. It creates the lookup desk utilizing linear interpolation for every main shade, with the 0–1 area divided into any variety of segments. For extra particulars, see this quick tutorial on making customized colormaps with Matplotlib.

Loading and prepping the illness knowledge

Subsequent, we load the CSV file into pandas and prep it for plotting. This file incorporates the incidence of measles (because the variety of circumstances per 100,000 folks) for every state (and the District of Columbia) by week from 1928 to 2003. We’ll have to convert the values to a numeric knowledge kind, mixture the info by 12 months, and reshape the DataFrame for plotting.

# Learn the csv file right into a DataFrame:
url = 'https://bit.ly/3F47ejX'
df_raw = pd.read_csv(url)

# Convert to numeric and mixture by 12 months:
df_raw.iloc[:, 2:] = (df_raw.iloc[:, 2:]
                      .apply(pd.to_numeric, 
                             errors='coerce'))

df = (df_raw.groupby('YEAR', as_index=False)
        .sum(min_count=1, numeric_only=True)
        .drop(columns=['WEEK']))

# Reshape the info for plotting:
df_melted = df.soften(id_vars='YEAR',
                    var_name='State',
                    value_name='Incidence')

df_pivot = df_melted.pivot_table(index='State',
                                 columns='YEAR',
                                 values='Incidence')

# Reverse the state order for plotting:
df_pivot = df_pivot[::-1]

Right here’s how the preliminary (uncooked) DataFrame seems, exhibiting the primary 5 rows and ten columns:

NaN values are represented by a splash (-). 

The ultimate df_pivot DataFrame is in vast format, the place every column represents a variable, and rows characterize distinctive entities:

Whereas plotting is usually carried out utilizing lengthy format knowledge, as within the df_raw DataFrame, pcolormesh() prefers vast format when making heatmaps. It is because heatmaps are inherently designed to show a 2D matrix-like construction, the place rows and columns characterize distinct classes. On this case, the ultimate plot will look very like the DataFrame, with states alongside the y-axis and years alongside the x-axis. Every cell of the heatmap might be coloured based mostly on the numerical values.

Dealing with lacking values 

The dataset incorporates a variety of lacking values. We’ll wish to distinguish these from 0 values within the heatmap by making a masks to establish and retailer these NaN values. Earlier than making use of this masks with NumPy, we’ll use Matplotlib’s Normalize() class to normalize the info. This fashion, we are able to straight examine the heatmap colours throughout states.

# Create a masks for NaN values:
nan_mask = df_pivot.isna()

# Normalize the info for a shared colormap:
norm = Normalize(df_pivot.min().min(), df_pivot.max().max())

# Apply normalization earlier than masking:
normalized_data = norm(df_pivot)

# Create masked array from normalized knowledge:
masked_data = np.ma.masked_array(normalized_data, masks=nan_mask)

Plotting the heatmap

The next code creates the heatmap. The center of it consists of the one line calling the pcolormesh() operate. Many of the relaxation ornaments the plot in order that it seems just like the WSJ heatmap (apart from the x, y, and colorbar labels, that are drastically improved in our model).

# Plot the info utilizing pcolormesh with a masked array:
multiplier = 0.22  # Modifications determine facet ratio
fig, ax = plt.subplots(figsize=(11, len(df_pivot.index) * multiplier))

states = df_pivot.index
years = df_pivot.columns

im = plt.pcolormesh(masked_data, cmap=custom_cmap, 
                    edgecolors='w', linewidth=0.5)

ax.set_title('Measles Incidence by State (1928-2002)', fontsize=16)

# Regulate x-axis ticks and labels to be centered:
every_other_year_indices = np.arange(0, len(years), 2) + 0.5
ax.set_xticks(every_other_year_indices)
ax.set_xticklabels(years[::2], rotation='vertical', fontsize=10)

# Regulate labels on y-axis:
ax.set_yticks(np.arange(len(states)) + 0.5)  # Heart ticks in cells
ax.set_yticklabels(states, fontsize=9)

# Add vertical line and label for vaccine date:
vaccine_year_index = listing(years).index(1963)
ax.axvline(x=vaccine_year_index, linestyle='--', 
           linewidth=1, shade='ok')
alaska_index = states.get_loc('ALASKA')
ax.textual content(vaccine_year_index, alaska_index, ' Vaccine', 
        ha='left', va='middle', fontweight='daring')

# Add a colorbar:
cbar = fig.colorbar(ScalarMappable(norm=norm, cmap=custom_cmap), 
                    ax=ax, orientation='horizontal', pad=0.1, 
                    label='Instances per 100,000')
cbar.ax.xaxis.set_ticks_position('backside')

plt.savefig('measles_pcolormesh_nan.png', dpi=600, bbox_inches='tight')
plt.present()

Right here’s the outcome:

It is a shut approximation of the WSJ heatmap, with what I take into account extra legible labels and higher separation of 0 and NaN (lacking knowledge) values. 

Makes use of for heatmaps

Heatmaps are extremely efficient at demonstrating how a blanket coverage or motion impacts a number of geographic areas over time. Due to their versatility, they are often tailored for different functions, comparable to monitoring:

• Air high quality index ranges in several cities earlier than and after the Clear Air Act
• Change in check scores for colleges or districts after insurance policies like No Baby Left Behind
• Unemployment charges for various areas after financial stimulus packages
• Product gross sales efficiency by area after native or nationwide advert campaigns

Among the many benefits of heatmaps is that they promote a number of evaluation methods. These embody:

Comparative Evaluation: simply examine traits throughout totally different classes ( states, colleges, areas, and so on.).

Temporal Traits: elegantly present how values change over time.

Sample Recognition: establish patterns and anomalies within the knowledge at a look.

Communication: Present a transparent and concise technique to talk advanced knowledge.

Heatmaps are an effective way to current a big-picture overview whereas preserving the info’s fine-scale granularity.