Correlation and Mutual Information

Correlation and mutual information are both ways to measure how much two things are related to each other.

Correlation is a measure of how much two variables move together. If one variable increases when the other increases, they are positively correlated. If one decreases when the other increases, they are negatively correlated. If there’s no consistent pattern, they are not correlated. Correlation values range from -1 (perfect negative correlation) to 1 (perfect positive correlation), with 0 indicating no correlation.

Mutual Information, on the other hand, is a bit more complex. It measures how much knowing the value of one variable reduces uncertainty about the value of the other. In other words, it tells us how much information about one variable is contained in the other. Mutual information values are always non-negative, with larger values indicating a stronger relationship.

While mutual information is a powerful tool for understanding the relationship between two variables within a given dataset or process, it’s not generally meaningful to compare mutual information values between different datasets or processes.

This is because the mutual information value is dependent on the specific distribution of the variables in question. Different processes or datasets can have different distributions, which can lead to different mutual information values even if the underlying relationships are similar.

Moreover, mutual information is measured in bits (or sometimes nats), which represent the reduction in uncertainty about one variable given knowledge of another. This reduction in uncertainty is specific to the variables and their distribution, and doesn’t necessarily translate to other contexts.

So, while mutual information can provide valuable insights into the relationships within a dataset, it should be used with caution when comparing across datasets or processes. It’s always important to consider the specific context and characteristics of the data you’re working with.

Let’s look at some random variables I’ve generated to see how this works out in an example

#packages
if (!require("infotheo")) {
  install.packages("infotheo")
}
if (!require("MASS")) {
  install.packages("MASS")
}
library(infotheo)
library(MASS)

# Set seed 
set.seed(123)
results <- data.frame(correlation = numeric(), mutual_information = numeric())

#Playing with correlations of interest (Playing COI)
for (correlation in seq(0, 1, by = 0.1)) {
  
  # bivariate normal dist
  data <- mvrnorm(1000, mu = c(0, 0), Sigma = matrix(c(1, correlation, correlation, 1), nrow = 2))
  
  mutual_information <- mutinformation(discretize(data[, 1]), discretize(data[, 2]))
  
  results <- rbind(results, c(correlation, mutual_information))
}

colnames(results) <- c("Correlation", "Mutual Information")
plot(results$Correlation, results$`Mutual Information`, 
     xlab = "Correlation", ylab = "Mutual Information", 
     main = "Correlation vs Mutual Information")

print(results)

When the correlation is 0 (meaning the variables are not correlated), the mutual information is also quite low, indicating that knowing the value of one variable doesn’t tell us much about the other.

As the correlation increases from 0 to 0.9, the mutual information also increases. This means that as the variables become more strongly correlated, knowing the value of one variable gives us more information about the other. I’ve also included correlation of 1 for completeness.

The mutual information increases more rapidly as the correlation gets higher. This suggests that even small increases in correlation can lead to large increases in mutual information when the correlation is already high.

Note that the mutual information for a correlation of .6 (~.24) is closer the mutual information of correlation 0.0 (~.03) than to correlation of .8 (~.49)

In summary, both correlation and mutual information are ways to measure the relationship between two variables, but they capture slightly different aspects of this relationship. While correlation measures the degree to which the variables move together, mutual information measures the amount of information they share.

A note for the curious: There are exceptions, but this is the rule