Heatmaps of Time Chain | Looking at the data science

In 2015, the Wall Street Journal (WSJ) It has been published by the most effective series of temperature showing the impact of vaccines from infectious diseases in the United States. This looks at the power of the gang policy to drive a wide change. You can view Heatmaps here.

Heatmaps are a variable device for data analysis. Their power to make analysis of comparisons, highlight a temporary tendency, and enabling pattern recognition makes them deal with complex information.

In this case Successful Success Success Project, we will use the Python's Mattplotlib library to restore Wsj's Card chart, showing how you can withstand Heathmaps and are carefully designed for colorful colors to influence teaching.

Data

Disease data from the University of Pittsburgh's Project Tycho. The organization is valid for national and international health facilities and investigators to make data easier to use to improve global health. The measles data is available under the Creative Commons Attribution license.

For convenience, I downloaded data from project Tycho data data in the CSV file and store it on this gear. Later, we will access your orderly.

Susesles Heatmap

We will use the matplotlib function () to create closer WSJ Mosjes HeatMap. While other libraries, such as seaborne, manifested, and HVplot, includes the activities provided by heat, this is built Easy Usewith many decisions issued. This makes it difficult to force their results to be the same Wsj Heatmap.

Exterior pcolormesh()Matplotlib's imshow() The function (of “Image Show”) can also produce heathmaps. This page pcolormesh The job, however, better grains have cells of the cell.

Here is a Heatmap Example made with imshow() that you compare with pcolormesh() results later. The main difference is the shortage of grins.

In 1963, the vaccine was a license and was released from America. Within five years, the incidence of the disease was reduced. In 2000, measles were considered shortened in the United States, and any new cases arrived outside the country. Note how visual visual transmits this “big picture” while maintaining a state level information. This is because of a smaller part in the flower.

The colored colors in seeing profusory. More than 80% of the color bar is covered by warm colors, and (light) blue is reserved for very small prices. This makes it easier to distinguish the pre-pre-assertime. White cells say Data lostrepresented by NAN (not number) Prices.

Compare the previous Heatmap to the formulated colored colorbar:

Black blue color is not only a plan, it's hard to look. And when it happened to see the outcome of the policy, a visible impact is more subtle than in revelation of colored colors. In contrast, it is easy to look at the higher amounts but by losing the total total.

Code

The following code was written in Juptetlab and was launched by the cell.

Import libraries

The first cell involves libraries we will need to complete the project. The Internet Search of the words in the library will result in installation instructions.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import LinearSegmentedColormap, Normalize
from matplotlib.cm import ScalarMappable
import pandas as pd

Creating a custom Colormap

The following code produces closely the ColorMap used by Wsj. I've used online Photo color selector Tool to identify important colors from their HeatMAP heat screenshot and modified this is based on the same colored study.

# Normalize RGB colors:
colors = ['#e7f0fa',  # lightest blue
          '#c9e2f6',  # light blue
          '#95cbee',  # blue
          '#0099dc',  # dark blue
          '#4ab04a',  # green
          '#ffd73e',  # yellow
          '#eec73a',  # yellow brown
          '#e29421',  # dark tan
          '#f05336',  # orange
          '#ce472e']  # red

# Create a list of positions for each color in the colormap:
positions = [0, 0.02, 0.03, 0.09, 0.1, 0.15, 0.25, 0.4, 0.5, 1]

# Create a LinearSegmentedColormap (continuous colors):
custom_cmap = LinearSegmentedColormap.from_list('custom_colormap', 
                                                list(zip(positions, 
                                                         colors)))

# Display a colorbar with the custom colormap:
fig, ax = plt.subplots(figsize=(6, 1))

plt.imshow([list(range(256))],
           cmap=custom_cmap, 
           aspect='auto', 
           vmin=0, vmax=255)

plt.xticks([]), plt.yticks([])
plt.show()

Here is the generic colorbar produced by code:

This code makes a progressive Colormap uses matplotlib's LinearSegmentedColormap() category. This class specifies Colormaps using Anchor points between RGB prices (a) be combined. That is, to create colormap materials based on the view tables using direct components. It creates a table of view using direct translation of each of the main color, with 0-1 domain divided into any category of categories. For more information, see this short lesson in making Colormaps customized with matplotlib.

Loading and Prepare for Data Data

Next, we upload a CSV file to pandas and prepare for planning. This file contains events of a mosque (As much as 100,000 people) in each kingdom (along with the Columbia District) Sunday from 1928 to 2003. We will need to convert prices into a form of numeric data, including a year's data, and restore planning data data.

# Read the csv file into a DataFrame:
url = '
df_raw = pd.read_csv(url)

# Convert to numeric and aggregate by year:
df_raw.iloc[:, 2:] = (df_raw.iloc[:, 2:]
                      .apply(pd.to_numeric, 
                             errors='coerce'))

df = (df_raw.groupby('YEAR', as_index=False)
        .sum(min_count=1, numeric_only=True)
        .drop(columns=['WEEK']))

# Reshape the data for plotting:
df_melted = df.melt(id_vars='YEAR',
                    var_name='State',
                    value_name='Incidence')

df_pivot = df_melted.pivot_table(index='State',
                                 columns='YEAR',
                                 values='Incidence')

# Reverse the state order for plotting:
df_pivot = df_pivot[::-1]

Here's how the first (green dataframe) looks, showing the first five lines and ten columns:

NaN Values ​​are represented by Dash (-).

Last df_pivot The DataFram is in A broad formatWhen each column represents flexibility, and the lines represent different structures:

While editing is usually done using A long format data, as in df_raw DATAFRAME, pcolormesh() You choose a broad format when making Heatmaps. This is because the Heathmaps are naturally designed to display a system such as the 2D such as 2D, where lines and column represents different categories. In this case, the last plan will look like a DataFrame, provinces and y-axis and ages there in X-Axis. Each HeatMAP ceremony will be the color based on number values.

Managing Missing Prices

The data contains a lot of missing amounts. We will want to divide this from 0 rates in the heat of a mask to identify and keep these NaN values. Before using this mask with nunpy, we will use matplotlib's Normalize() class to zoo Information. In this way, we can directly compare the HEATMAP colors available.

# Create a mask for NaN values:
nan_mask = df_pivot.isna()

# Normalize the data for a shared colormap:
norm = Normalize(df_pivot.min().min(), df_pivot.max().max())

# Apply normalization before masking:
normalized_data = norm(df_pivot)

# Create masked array from normalized data:
masked_data = np.ma.masked_array(normalized_data, mask=nan_mask)

You plan to burn

The following code creates heat. Her heart contains one expensive line pcolormesh() work. Most resters decorate conspiracy to look like Wsj Heatmap (without X, Y, and Colorsbars, are very developed in our version).

# Plot the data using pcolormesh with a masked array:
multiplier = 0.22  # Changes figure aspect ratio
fig, ax = plt.subplots(figsize=(11, len(df_pivot.index) * multiplier))

states = df_pivot.index
years = df_pivot.columns

im = plt.pcolormesh(masked_data, cmap=custom_cmap, 
                    edgecolors='w', linewidth=0.5)

ax.set_title('Measles Incidence by State (1928-2002)', fontsize=16)

# Adjust x-axis ticks and labels to be centered:
every_other_year_indices = np.arange(0, len(years), 2) + 0.5
ax.set_xticks(every_other_year_indices)
ax.set_xticklabels(years[::2], rotation='vertical', fontsize=10)

# Adjust labels on y-axis:
ax.set_yticks(np.arange(len(states)) + 0.5)  # Center ticks in cells
ax.set_yticklabels(states, fontsize=9)

# Add vertical line and label for vaccine date:
vaccine_year_index = list(years).index(1963)
ax.axvline(x=vaccine_year_index, linestyle='--', 
           linewidth=1, color='k')
alaska_index = states.get_loc('ALASKA')
ax.text(vaccine_year_index, alaska_index, ' Vaccine', 
        ha='left', va='center', fontweight='bold')

# Add a colorbar:
cbar = fig.colorbar(ScalarMappable(norm=norm, cmap=custom_cmap), 
                    ax=ax, orientation='horizontal', pad=0.1, 
                    label='Cases per 100,000')
cbar.ax.xaxis.set_ticks_position('bottom')

plt.savefig('measles_pcolormesh_nan.png', dpi=600, bbox_inches='tight')
plt.show()

Here is the result:

This is closest to the Wsj Heatmap, and what I look like as a more accessible lab and better classification of 0 and NaN (Missing data values).

Using Heatmaps

Heatmaps are very efficient in showing that a clothing policy or action affects many local circuits later. Due to their various fluctuations, they can be changed by other purposes, such as tracking:

• Air Quality Index levels in different cities before and after Law to clean the air
• Change in test score in schools or regions after policies such as No child is left behind
• Unemployment prices in different districts after the financing of economy
• Product Retail Workment After Location Campaigns or Country

Among the Heatmaps heat benefits that promote many analysis techniques. This includes:

Comparison comparisons: Compare easily styles in all different categories (circuits, schools, districts, etc.).

Temporary styles: Reflect on how prices changes later.

Recognition of pattern: Identify patterns and animomalies on the data off.

Communication: Provide a clear and brief approach to communication with complex data.

Heatmaps is a great way to introduce looking for all of the big big