Extracting Text from PDF

Introduction

This document is published in the public domain on my RPubs account.

This is a demonstration of using the R language pdftools and stringr packages to scrape data from PDF files. I produced this example after reading a Revolutions blog which describes an interesting application wherein locations of medical dispensaries were scrapped from a PDF on the internet and used to plot points on a map.

I have seen good examples of where we could use PDF scraping tools to extract data from PDF files to use in scripts. In the past, I have been required to do this manually (from contract documentation) in order generate KKS code look-up tables which we are using in PFA analysis. Ideally, I would automate this manual process so that I:

• Do not make mistakes.
• Perform the data extraction more quickly.
• Can re-use analyses on similar PDF documents.
• Can record explicitly what I did in an analysis (i.e. enhance reproducibility).

In this example, I will demonstrate how to extract KKS codes which are tabulated in a PDF document. A document published by Siemens is publicly available on the web which I use regularly as a reference for the KKS code system. The document can be accessed by clicking here. My objective will be to write a script in R to extract the table on page 9 and create a look-up table. The process is:

1. Download the document directly from the web.
2. Read the entire contents.
3. Select page 9 which contains the table of interest.
4. Parse the text and extract the lines containing useful information.
5. Process the text into a look-up table.
6. Demonstrate usage.

Download and Read File

The file is downloaded and then read using the pdftools package. Each page is loaded into an element of a vector so in order to access page 9, I simply index that element.

library(pdftools)
my_url <- paste0("http://diskuse.elektrika.cz/index.php?action=dlattach;", 
                 "topic=18673.0;attach=10786")
my_file <- "KKS-Indentification-System.pdf"
download.file(my_url, my_file)

trying URL 'http://diskuse.elektrika.cz/index.php?action=dlattach;topic=18673.0;attach=10786'
Content type 'application/octet-stream' length 280693 bytes (274 KB)
==================================================
downloaded 274 KB

my_page <- pdf_text(my_file)[9]
print(my_page)

[1] "KKS Identification\n    Power Plants\n                   System for\n                                                    Function Key\n     Function Key, Main Groups\nA    Grid and distribution systems\nB    Power transmission and auxiliary power supply\nC    Instrumentation and control equipment\nE    Fuel supply and residues disposal\nG    Water supply and disposal\nH    Heat generation\nL    Steam, water, gas cycles\nM    Main machine sets\nN    Process energy/fluid supply for external users\n     (e.g. district heating)\nP    Cooling water systems\nQ    Auxiliary systems\nR    Gas generation and treatment\nS    Ancillary systems\nU    Structures\nW    Renewable energy plants\nX    Heavy machinery (not main machine sets)\n     (e.g. emergency diesel and generator sets)\n                              2\n"

Parse and Process Text

The above code has given access to the raw text on the page. Note that \n is a special character that denotes a newline. It is now necessary to extract the useful text from the table and arrange it appropriately into a look-up table. I use some text processing tools from the stringr package to extract the relevant text. NB: This could be done with base R but I prefer the syntax available from the stringr package.

library(stringr)
my_table <- my_page %>% 
    str_split("\n") %>% 
    .[[1]] %>% 
    str_subset("^[A-Z]\\s")
print(my_table)

 [1] "A    Grid and distribution systems"                 
 [2] "B    Power transmission and auxiliary power supply" 
 [3] "C    Instrumentation and control equipment"         
 [4] "E    Fuel supply and residues disposal"             
 [5] "G    Water supply and disposal"                     
 [6] "H    Heat generation"                               
 [7] "L    Steam, water, gas cycles"                      
 [8] "M    Main machine sets"                             
 [9] "N    Process energy/fluid supply for external users"
[10] "P    Cooling water systems"                         
[11] "Q    Auxiliary systems"                             
[12] "R    Gas generation and treatment"                  
[13] "S    Ancillary systems"                             
[14] "U    Structures"                                    
[15] "W    Renewable energy plants"                       
[16] "X    Heavy machinery (not main machine sets)"       

The processing steps are:

• Split the text by new lines.
• Extract the lines which begin with a single capital letter, A-Z, followed by a single space, for which I use a regular expression.

Make Lookup Table

So far, the relevant lines on the page have been read as a character vector which contains the KKS code and description as one item of text. However, our task is to be able to ‘look-up’ a KKS code and return a description. We achieve this by splitting the each element of the character vector into a ‘key’ and ‘value’ pair. The key will be the KKS code and value will be the description.

library(dplyr)
my_lookup <- my_table %>%
    str_split_fixed("\\s{2,}", n = 2) %>% 
    as_data_frame() %>% 
    rename(Code = V1, Description = V2)
print(my_lookup)

I have achieved this simply by splitting on the first instance where this is more than one space between word boundaries.

Demonstration

Now we can look-up codes programmatically, for example:

• Look-up description of KKS code A.
• Look-up descriptions of KKS codes E, F and G.

my_lookup %>% right_join(data_frame(Code = 'A'))

Joining, by = "Code"

my_lookup %>% right_join(data_frame(Code = c('E', 'F', 'G')))

Joining, by = "Code"

If the tag code does not exist, an NA is gracefully returned.

Conclusions

It would be trivial to continue reading pages in this fashion. There are very few lines of code required to extract, parse and process the data. I will explore more extensive usage of PDF scraping as opportunities arise.