Data Sonification

This is a basic data sonification using R. The process is as follows:

While this initial foray into sonification was successful - sound was produced that matched the original data, there are a number of issues with the actual process. The most crucial of these is automation - many of the mappings are done by hand. This is an inherent problem in the nascent field of sonification, which ought to be addressed in future research.

Much like visualization is a way to perceive data visually, sonification is a way to perceive data aurally. Any sonification must, first and foremost, reflect the properties and relations in the input data, so sonification itself is not intrinsically a musical process. Nevertheless, the input data could be chosen for musical reasons.

There is a great potential for overlap between the fields of sonification and music theory. Within the last century, music theorists amassed an extensive literature regarding the gestalt properties of sound and their manifestations in music. An interdisciplinary approach to sonification will meld cognitive science, music theory, and physical sound to fully realize the acoustic possibilities of auditory representations of data.

More about the field of sonification can be read here, and an open access handbook that outlines the approaches to the field is available here.

View individual scripts and hear the audio here.


Data Processing

The data for this sonification is very simple - 12 random normal variables in three clusters.

Generate some simple data & plot it 

set.seed(1234)
x <- rnorm(12, mean = rep(1:3, each = 4), sd = 0.2)
y <- rnorm(12, mean = rep(c(1, 2, 1), each = 4), sd = 0.2)
plot(x, y, col = "blue", pch = 19, cex = 2)
text(x + 0.05, y + 0.05, labels = as.character(1:12))

Make a data frame out of the data and find its k-means 

dataFrame <- data.frame(x, y)
kmeansObj <- kmeans(dataFrame, centers = 3)

Make a dendrogram out of the DataFrame 

distxy <- dist(dataFrame)
hc <- hclust(distxy)
plot(hc)

Plot the clustered data 

plot(x, y, col=kmeansObj$cluster, pch=19, cex=2)
points(kmeansObj$centers, col=c("black","red","green"), pch=3, cex=3, lwd=3)


Scale Process

Map values in the y-axis to tonal functions via clustered groups, given a predefined scale (C Major).

Make a dataframe that attaches dataFrame to kmeansObj$cluster 

dataFrame$cluster <- as.factor(kmeansObj$cluster)

Make a vector that defines the scale 

major <- c(0, 2, 4, 5, 7, 9, 11)

Make chord values from the scale

Make chord values from the scale

scaleFunc <- list(tonic = c(major[1], major[3], major[5]), predominant =
                          c(major[4], major[6], 12 + major[2]), dominant =
                          c(major[5], major[7], 12 + major[2]))

Assign a chord value to each cluster

Subset the max and min for each cluster 

dataFrame$pitch <- 0
dataMin <- dataFrame[dataFrame$y == ave(dataFrame$y, dataFrame$cluster, FUN=min), ]
dataMax <- dataFrame[dataFrame$y == ave(dataFrame$y, dataFrame$cluster, FUN=max), ]

Find the difference in y values, divide each by three, and use this to determine the scale-step cutoff this part is really messy 

cutOff <- (dataMax$y - dataMin$y)/3

c3 <- dataFrame[dataFrame$cluster == 3, ]
for (i in 1:nrow(c3)) {
if (c3$y[i] <= (dataMin$y[1] + cutOff[1])) {
        c3$pitch[i] = scaleFunc$tonic[1]
} else if (c3$y[i] <= (dataMin$y[1] + (cutOff[1])*2)) {
        c3$pitch[i] = scaleFunc$tonic[2]
} else if (c3$y[i] <= (dataMin$y[1] + (cutOff[1])*3)) {
        c3$pitch[i] = scaleFunc$tonic[3]
}
}

c1 <- dataFrame[dataFrame$cluster == 1, ]
for (i in 1:nrow(c1)) {
        if (c1$y[i] <= (dataMin$y[2] + cutOff[2])) {
                c1$pitch[i] = scaleFunc$predominant[1]
        } else if (c1$y[i] <= (dataMin$y[2] + (cutOff[2])*2)) {
                c3$pitch[i] = scaleFunc$predominant[2]
        } else if (c1$y[i] <= (dataMin$y[2] + (cutOff[2])*3)) {
                c1$pitch[i] = scaleFunc$predominant[3]
        }
}

c2 <- dataFrame[dataFrame$cluster == 2, ]
for (i in 1:nrow(c2)) {
        if (c2$y[i] <= (dataMin$y[3] + cutOff[3])) {
                c2$pitch[i] = scaleFunc$dominant[1]
        } else if (c2$y[i] <= (dataMin$y[3] + (cutOff[3])*2)) {
                c2$pitch[i] = scaleFunc$dominant[2]
        } else if (c2$y[i] <= (dataMin$y[3] + (cutOff[3])*3)) {
                c2$pitch[i] = scaleFunc$dominant[3]
        }
}

dataFrame <- rbind(c3, c1, c2)

Look at dataFrame 

dataFrame
##            x         y cluster pitch
## 1  0.7585869 0.8447492       3     0
## 2  1.0554858 1.0128918       3     4
## 3  1.2168882 1.1918988       3     7
## 4  0.5308605 0.9779429       3     4
## 5  2.0858249 1.8977981       1     5
## 6  2.1012112 1.8177609       1     5
## 7  1.8850520 1.8325657       1     5
## 8  1.8906736 2.4831670       1    14
## 9  2.8871096 1.0268176       2    11
## 10 2.8219924 0.9018628       2     7
## 11 2.9045615 0.9118904       2     7
## 12 2.8003227 1.0919179       2    14


Rhythm Process

Map x-value to duration.

Maximum x value ~ Duration 

duration <- 960
max.x <- max(dataFrame$x, na.rm = TRUE)
mult <- (duration - 50)/max.x

Make a duration column in dataFrame - x*mult 

dataFrame$time <- x*mult
dataFrame <- dataFrame[order(dataFrame$time), ]

Note cutoffs per cluster - min(cluster$x)-50 

dataMax.x <- dataFrame[dataFrame$time == ave(dataFrame$time, dataFrame$cluster, FUN=max), ]
cutoff3 <- dataMax.x[1, ]$time + 50
cutoff1 <- dataMax.x[2, ]$time + 50
cutoff2 <- duration


Construct CSV

Construct a CSV to be converted into a midi file

require(plyr)
## Loading required package: plyr
setwd("~/Documents/Sonification")

Begin with the metadata

0, 0, Header, format, nTracks, division

The first record of a CSV MIDI file is always the Header record. Parameters are format: the MIDI file type (0, 1, or 2), nTracks: the number of tracks in the file, and division: the number of clock pulses per quarter note. The Track and Time fields are always zero.

division <- 480
header <- list(0, 0, "Header", 1, 2, division)
header <- as.data.frame(header)
colnames(header) <- c("track", "time", "event", "channel", "pitch", "velocity")

Track, 0, Start_track

A Start_track record marks the start of a new track, with the Track field giving the track number. All records between the Start_track record and the matching End_track will have the same Track field.

Start_track <- list(1, 0, "Start_track")
Start_track <- as.data.frame(Start_track)
colnames(Start_track) <- c("track", "time", "event")

Track, Time, Title_t, Text

The Text specifies the title of the track or sequence. The first Title meta-event in a type 0 MIDI file, or in the first track of a type 1 file gives the name of the work. Subsequent Title meta-events in other tracks give the names of those tracks.

title.track <- "sonification"
Title_t <- list(1, 0, "Title_t", title.track)
Title_t <- as.data.frame(Title_t)
colnames(Title_t) <- c("track", "time", "event", "channel")

Track, Time, Text_t, Text

This meta-event supplies an arbitrary Text string tagged to the Track and Time. It can be used for textual information which doesn’t fall into one of the more specific categories given above.

text.track <- "This is a test midi that is a sonification of arbitrary data."
Text_t <- list(1, 0, "Text_t", text.track)
Text_t <- as.data.frame(Text_t)
colnames(Text_t) <- c("track", "time", "event", "channel")

Track, Time, Time_signature, Num, Denom, Click, NotesQ

The time signature, metronome click rate, and number of 32nd notes per MIDI quarter note (24 MIDI clock times) are given by the numeric arguments. Num gives the numerator of the time signature as specified on sheet music. Denom specifies the denominator as a negative power of two, for example 2 for a quarter note, 3 for an eighth note, etc. Click gives the number of MIDI clocks per metronome click, and NotesQ the number of 32nd notes in the nominal MIDI quarter note time of 24 clocks (8 for the default MIDI quarter note definition).

Time_signature <- list(1, 0, "Time_signature", 4, 2, 24, 8)
Time_signature <- as.data.frame(Time_signature)
colnames(Time_signature) <- c("track", "time", "event", "channel", "pitch", "velocity",
                              "V7")

Track, Time, Tempo, Number

The tempo is specified as the Number of microseconds per quarter note, between 1 and 16777215. A value of 500000 corresponds to 120 quarter notes (“beats”) per minute. To convert beats per minute to a Tempo value, take the quotient from dividing 60,000,000 by the beats per minute.

Tempo <- list(1, 0, "Tempo", 500000)
Tempo <- as.data.frame(Tempo)
colnames(Tempo) <- c("track", "time", "event", "channel")

Track, Time, End_track

An End_track marks the end of events for the specified Track. The Time field gives the total duration of the track, which will be identical to the Time in the last event before the End_track.

End_track <- list(1, 0, "End_track")
End_track <- as.data.frame(End_track)
colnames(End_track) <- c("track", "time", "event")

Channel Events 

Start_track2 <-list(2, 0, "Start_track")
Start_track2 <- as.data.frame(Start_track2)
colnames(Start_track2) <- c("track", "time", "event")

Instrument_name <- list(2, 0, "Instrument_name_t", "Acoustic Grand Piano")
Instrument_name <- as.data.frame(Instrument_name)
colnames(Instrument_name) <- c("track", "time", "event", "channel")

Track, Time, Program_c, Channel, Program_num

Switch the specified Channel to program (patch) Program_num, which must be between 0 and 127. The program or patch selects which instrument and associated settings that channel will emulate. The General MIDI specification provides a standard set of instruments, but synthesisers are free to implement other sets of instruments and many permit the user to create custom patches and assign them to program numbers.

Program_c <- list(2, 0, "Program_c", 1, 1)
Program_c <- as.data.frame(Program_c)
colnames(Program_c) <- c("track", "time", "event", "channel", "pitch")

Construct a dataframe of note_on events 

track <- 2
time <- dataFrame$time
event <- "Note_on_c"
channel <- 1
pitch <- dataFrame$pitch + 60
velocity <- 127
on_events <- data.frame(track, time, event, channel, pitch, velocity, 
                        check.names = FALSE)

Construct dataframes of note_off events 

off_event <- "Note_off_c"
off_events1 <- data.frame(track, cutoff3, off_event, channel, scaleFunc$tonic + 60,
                         velocity, row.names=NULL)
names(off_events1) <- c("track", "time", "event", "channel", "pitch", "velocity")
off_events2 <- data.frame(track, cutoff1, off_event, channel, 
                         scaleFunc$predominant + 60, velocity)
names(off_events2) <- c("track", "time", "event", "channel", "pitch", "velocity")
off_events3 <- data.frame(track, cutoff2, off_event, channel, 
                          scaleFunc$dominant + 60, velocity)
names(off_events3) <- c("track", "time", "event", "channel", "pitch", "velocity")
off_events <- rbind(off_events1, off_events2, off_events3)

Combine note_on and note_off events 

events <- rbind(on_events, off_events)
events <- events[order(events$time), ]

Track, Time, End_track

An End_track marks the end of events for the specified Track. The Time field gives the total duration of the track, which will be identical to the Time in the last event before the End_track.

End_track2 <- list(2, duration, "End_track")
End_track2 <- as.data.frame(End_track2)
colnames(End_track2) <- c("track", "time", "event")

0, 0, End_of_file

The last record in a CSV MIDI file is always an End_of_file record. Its Track and Time fields are always zero.

End_of_file <- list(0, 0, "End_of_file")
End_of_file <- as.data.frame(End_of_file)
colnames(End_of_file) <- c("track", "time", "event")


csv <- rbind.fill(header, Start_track, Title_t, Text_t, Time_signature, Tempo,
                  End_track, Start_track2, Instrument_name, Program_c, events,
                  End_track2, End_of_file)
colnames(csv) <- NULL
na.omit(csv)
##   NA NA             NA NA NA NA NA
## 5  1  0 Time_signature  4  2 24  8
write.csv(csv, file="testmidi.csv", quote=FALSE, na="", row.names=FALSE)

Conclusion

As mentioned in the introduction, sonification necessitates a greater degree of automation. Nevertheless, the assignment tonal functions to hierarchical clusters is a new and useful addition to the field, especially when exploring the multitude of possibilities in sonic space via alternate scales and tuning systems.