De-duplication analysis

When reading Appendix B - Managing Heritrix Harvest Templates from the NetarchiveSuite I came across this text: “Note that only non-text objects are examined by this module, i.e. where the mimetype of the object does not match”^text/." on how the suite decides whether a harvested document could be de-duplicated. So how well can we trust the MIME type responded from the web server serving the web document? This small analyis can maybe give an idea of an answer to that question.

During an earlier project I extracted MIME type information from circa 250 million documents from the Netarkivet. It should be stated that the data is extracted from circa. 80.000 arbitrary selected ARC files. This same data set has been explored in the A Weekend With Nanite blog post

Let’s look at the data using R/R Studio. First the data is read into the workbench.

d<-read.table("all-parts",
  header=FALSE, 
  sep='\t', 
    comment.char="", 
    col.names=c("?","http","droid","tika","tikap","year","count"), 
    fill=TRUE,
    na.string=c("","Exception"))

## Warning: EOF within quoted string

d_df <- tbl_df(d)

We only want to look at documents that by the HTTP server was reported as text documents.

texts <- filter(d_df,substr(http,1,5)=="text/")

How many documents were actually reported as text objects?

summarise(texts,counts=sum(count))

## Source: local data frame [1 x 1]
## 
##      counts
## 1 188921319

Now remove everything that is identified as text documents by a file format detection tool (Apache Tika) and print the top-15 of these

texts.tika <- texts %>%
  group_by(tika) %>%
  summarise(counts=sum(count)) %>%
  arrange(desc(counts))

texts.tika.not <- filter(texts.tika,!substr(tika,1,5)=="text/")
top15<-head(texts.tika.not,15)
top15

## Source: local data frame [15 x 2]
## 
##                             tika   counts
## 1          application/xhtml+xml 54831245
## 2       application/octet-stream 10846241
## 3            application/rss+xml   757920
## 4                application/xml   518373
## 5                     image/jpeg   467580
## 6         application/javascript   380341
## 7                      image/gif   124020
## 8             application/x-gzip    77549
## 9                      image/png    72809
## 10      application/x-msdownload    55502
## 11 application/x-shockwave-flash    45239
## 12                video/x-ms-wmv    44625
## 13      image/vnd.microsoft.icon    33050
## 14           application/rdf+xml    23992
## 15                audio/x-ms-wma    12424

summarise(texts.tika.not,counts=sum(counts))

## Source: local data frame [1 x 1]
## 
##     counts
## 1 68399094

Let’s look at that in a visual way

So, it looks like we miss the oportunity of de-deuplicating a few percent. A few percent of 500TB is tens of terabytes.

To be Continued…