Note: necessary packages are: tidyverse, ruODK, leaflet

Setting up the ruODK connection

Note: obviously you need to de-comment the pw = "your_password" field. I am not showing it for obvious reasons.

ruODK::ru_setup(
    url = "https://odk.idradata.org",
    svc = "https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc",
    un = "mfilipski@uga.edu",
    # pw = ".....",
    tz = "America/New_York",
    pid = "1", # Default project
    fid = "odk_training_lesson_BR", # Default form
    verbose = TRUE # Shows all the params, no need here
)
## <ruODK settings>
##   Default ODK Central Project ID:  1 
##   Default ODK Central Form ID:  odk_training_lesson_BR 
##   Default ODK Central URL:  https://odk.idradata.org 
##   Default ODK Central Username:  mfilipski@uga.edu 
##   Default ODK Central Password: run ruODK::get_default_pw() to show 
##   Default Time Zone:  America/New_York 
##   Default ODK Central Version:  0.8 
##   Test ODK Central Project ID:  
##   Test ODK Central Form ID:  
##   Test ODK Central Form ID (ZIP tests):  
##   Test ODK Central Form ID (Attachment tests):  
##   Test ODK Central Form ID (Parsing tests):  
##   Test ODK Central URL:  
##   Test ODK Central Username:  
##   Test ODK Central Password: run ruODK::get_test_pw() to show 
##   Test ODK Central Version:  
##   Verbose messages: TRUE

Getting projects and forms

These commands extract the meta-data

proj <- project_list()
proj_detail <- project_detail() # this is a tad different than previous
form_list <- form_list()

Contents (metadata) for the default project

This is some info about the project:

knitr::kable(proj)
id name forms app_users created_at updated_at last_submission archived
1 Default Project 3 5 2020-03-28 16:50:06 NA 2020-04-29 18:41:12 FALSE

Detail of forms in the project

These are the forms we have:

knitr::kable(form_list)
name fid version state submissions created_at created_by_id created_by updated_at last_submission hash
IDRA-testform-maps IDRA-testform-maps 2020043001 open 1 2020-04-29 18:33:14 17 Edward Whitney 2020-04-30 12:16:19 2020-04-29 18:41:12 97d03175982b896ed5232e63531e7557
demo odk_training_lesson_BR 1 open 6 2020-04-13 21:11:53 5 2020-04-29 18:33:31 2020-04-25 00:03:51 7edf960e1b58643dce9374e050cae24a
odk_training_lesson_BR_ed2 odk_training_lesson_BR_ed2 2020042703 open 2 2020-04-27 20:23:30 17 Edward Whitney 2020-04-29 18:33:31 2020-04-28 13:55:06 5f7d3b8fcf37d02f34de4ab92ab3e91a

Switching between forms:

This is how we can get at the form ids. Theoretically you can just loop over this in analysis, though of course the forms are not likely to have very similar variable names etc.

form_list[[2]][1] # First one
## [1] "IDRA-testform-maps"
form_list[[2]][2] # Second one
## [1] "odk_training_lesson_BR"

Getting contents from a form

Demo form

BR is already setup as the default form ### Extract and display all the data in the form: This extracts the data:

# ruODK::ru_setup(fid = "odk_training_lesson_BR")  # not necessary since already default
data_f1 <- ruODK::odata_submission_get()

This is the data:

knitr::kable(data_f1)
id dm001 br003 br005 r2_count system_submission_date system_submitter_id system_submitter_name system_attachments_present system_attachments_expected g1_r1_count g1_r1_odata_navigation_link r2_odata_navigation_link meta_instance_id br006 br007 odata_context
uuid:9f7a28fd-0e6c-4697-929d-013fd836419e 2 0 3 1 2020-04-25T04:03:51.216Z 19 Ed 0 0 2 Submissions(‘uuid:9f7a28fd-0e6c-4697-929d-013fd836419e’)/G1/R1 Submissions(‘uuid:9f7a28fd-0e6c-4697-929d-013fd836419e’)/R2 uuid:9f7a28fd-0e6c-4697-929d-013fd836419e NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 1 1 NA 1 2020-04-15T03:36:58.486Z 9 Default Project User 0 0 1 Submissions(‘uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13’)/G1/R1 Submissions(‘uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13’)/R2 uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 3 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 3 1 NA 1 2020-04-14T01:31:46.302Z 9 Default Project User 0 0 3 Submissions(‘uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37’)/G1/R1 Submissions(‘uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37’)/R2 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 956 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:09a89415-434a-4004-b611-188513e57b6b 1 2 NA 1 2020-04-14T01:28:52.729Z 9 Default Project User 0 0 1 Submissions(‘uuid:09a89415-434a-4004-b611-188513e57b6b’)/G1/R1 Submissions(‘uuid:09a89415-434a-4004-b611-188513e57b6b’)/R2 uuid:09a89415-434a-4004-b611-188513e57b6b NA 6568 https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c 1 1 NA 1 2020-04-14T01:28:52.222Z 9 Default Project User 0 0 1 Submissions(‘uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c’)/G1/R1 Submissions(‘uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c’)/R2 uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c 32 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:97193853-6939-4d7b-9e64-8aa784528360 1 1 NA 1 2020-04-14T01:21:09.922Z 9 Default Project User 0 0 1 Submissions(‘uuid:97193853-6939-4d7b-9e64-8aa784528360’)/G1/R1 Submissions(‘uuid:97193853-6939-4d7b-9e64-8aa784528360’)/R2 uuid:97193853-6939-4d7b-9e64-8aa784528360 6446 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions

Now the form is just a dataframe

You can do what you want with the data

Ex: Display specific variables

# dm001 variable (number of people in household?):
data_f1$dm001
## [1] 2 1 3 1 1 1
# Submitter name: 
data_f1$system_submitter_name
## [1] "Ed"                   "Default Project User" "Default Project User"
## [4] "Default Project User" "Default Project User" "Default Project User"

Make a plot

I’m not sure what the dm001 variable actually is, but you get the idea:
This is the variable dm001 plotted against order of submission (reversed), colored by who made the submission

data_f1$nsub = rownames(data_f1) ## Nicer ID variable than the system one
g_dm001_f1 <- ggplot(data_f1, aes(x=nsub, y=dm001)) + 
    geom_point(aes(color=system_submitter_name)) + 
    labs(title ="Maybe: Number of people in HH (dm001)?")
g_dm001_f1

Second form

Extract second form data

You can just switch the form and extract the data:

ruODK::ru_setup(fid = "odk_training_lesson_BR_ed2")
data_f2 <- ruODK::odata_submission_get()

This is form 2 data:

knitr::kable(data_f2)
id dm001 dm001b br003 br007 r2_count system_submission_date system_submitter_id system_submitter_name system_attachments_present system_attachments_expected g1_r1_count g1_r1_odata_navigation_link r2_odata_navigation_link meta_instance_id odata_context
uuid:83f800f2-b21c-4100-a708-6db31acdcc14 3 2 2 8 1 2020-04-28T17:55:06.605Z 19 Ed 0 0 3 Submissions(‘uuid:83f800f2-b21c-4100-a708-6db31acdcc14’)/G1/R1 Submissions(‘uuid:83f800f2-b21c-4100-a708-6db31acdcc14’)/R2 uuid:83f800f2-b21c-4100-a708-6db31acdcc14 https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR_ed2.svc/$metadata#Submissions
uuid:a493d820-b5f3-4a88-a7c2-056f68c97adb 2 NA 1 NA 1 2020-04-28T00:25:24.381Z 19 Ed 0 0 2 Submissions(‘uuid:a493d820-b5f3-4a88-a7c2-056f68c97adb’)/G1/R1 Submissions(‘uuid:a493d820-b5f3-4a88-a7c2-056f68c97adb’)/R2 uuid:a493d820-b5f3-4a88-a7c2-056f68c97adb https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR_ed2.svc/$metadata#Submissions

Plot something else

Example: the timing of submissions

# Extract just the data: 
data_f1$subtime = as.Date(data_f1$system_submission_date)
# Make the plot:
g_subtime_f1 <- ggplot(data_f1, aes(x=nsub, y=subtime)) + 
    geom_point() + 
    geom_label(aes(label=system_submitter_name), nudge_y = 1)
g_subtime_f1

Making a map

This uses the latest form.
Here’s a map of the submissions:

ruODK::ru_setup(fid = "IDRA-testform-maps") 
data_f3 <- ruODK::odata_submission_get()
data_f3$longitude <- data_f3$loc_coordinates_6
data_f3$latitude <- data_f3$loc_coordinates_7
leaflet(data_f3) %>% addTiles() %>% addMarkers()

Getting the REPEATS (i.e Sub-tables)

First of all, let’s list the repeat tables in that beer form with 6 submissions

form_to_load = "odk_training_lesson_BR" # Just making the code more generic for later
ruODK::ru_setup(fid = form_to_load)
form_tables <- ruODK::odata_service_get()

The repeats are called “tables” for odata

knitr::kable(form_tables)
name kind url
Submissions EntitySet Submissions
Submissions.G1.R1 EntitySet Submissions.G1.R1
Submissions.R2 EntitySet Submissions.R2

Now we can extract each of those with an option:

maintab <- odata_submission_get() # default: main data table
# maintab <- odata_submission_get(table = form_tables$url[1]) # same, but explicitly specifying table
knitr::kable(maintab)
id dm001 br003 br005 r2_count system_submission_date system_submitter_id system_submitter_name system_attachments_present system_attachments_expected g1_r1_count g1_r1_odata_navigation_link r2_odata_navigation_link meta_instance_id br006 br007 odata_context
uuid:9f7a28fd-0e6c-4697-929d-013fd836419e 2 0 3 1 2020-04-25T04:03:51.216Z 19 Ed 0 0 2 Submissions(‘uuid:9f7a28fd-0e6c-4697-929d-013fd836419e’)/G1/R1 Submissions(‘uuid:9f7a28fd-0e6c-4697-929d-013fd836419e’)/R2 uuid:9f7a28fd-0e6c-4697-929d-013fd836419e NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 1 1 NA 1 2020-04-15T03:36:58.486Z 9 Default Project User 0 0 1 Submissions(‘uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13’)/G1/R1 Submissions(‘uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13’)/R2 uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 3 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 3 1 NA 1 2020-04-14T01:31:46.302Z 9 Default Project User 0 0 3 Submissions(‘uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37’)/G1/R1 Submissions(‘uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37’)/R2 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 956 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:09a89415-434a-4004-b611-188513e57b6b 1 2 NA 1 2020-04-14T01:28:52.729Z 9 Default Project User 0 0 1 Submissions(‘uuid:09a89415-434a-4004-b611-188513e57b6b’)/G1/R1 Submissions(‘uuid:09a89415-434a-4004-b611-188513e57b6b’)/R2 uuid:09a89415-434a-4004-b611-188513e57b6b NA 6568 https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c 1 1 NA 1 2020-04-14T01:28:52.222Z 9 Default Project User 0 0 1 Submissions(‘uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c’)/G1/R1 Submissions(‘uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c’)/R2 uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c 32 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions
uuid:97193853-6939-4d7b-9e64-8aa784528360 1 1 NA 1 2020-04-14T01:21:09.922Z 9 Default Project User 0 0 1 Submissions(‘uuid:97193853-6939-4d7b-9e64-8aa784528360’)/G1/R1 Submissions(‘uuid:97193853-6939-4d7b-9e64-8aa784528360’)/R2 uuid:97193853-6939-4d7b-9e64-8aa784528360 6446 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions

Or we can eactract any other subtable. Two “ids” here" - “id” = individual id, - “submissions_id” = the one you can match back to maintab

subtab1 <- odata_submission_get(table = form_tables$url[2]) # sub-table 1
subtab2 <- odata_submission_get(table = form_tables$url[3]) # sub-table 1
knitr::kable(subtab1)
id submissions_id dm002 dm003 dm004 dm005 br001 br002 odata_context
7497124e52e5d3c6407bc33e314aad57f06d7b87 uuid:9f7a28fd-0e6c-4697-929d-013fd836419e R 1 5 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
0a297afc9773a1c443b440eedbd74e04463704d9 uuid:9f7a28fd-0e6c-4697-929d-013fd836419e T 2 2 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
e5ddb498cfa90c60908b2596216c88da457b726d uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 Hsue 2 1 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
18a5ce880ceff15a13e21a8993c1318e252bbb7e uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 Person 1 2 3 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
3e876b4b9e37079a21ff144e71163661071596f1 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 Person 2 2 1 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
b6633f2b9759711ef6bb4aead87b5164e15ac677 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 Person 3 2 60 6 1 26 https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
06f48f4f5a1a977d63c508bb5f833e935b0ba40e uuid:09a89415-434a-4004-b611-188513e57b6b Hi devon 2 3 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
49b8a700472b575901ede3fba3c86251f4d2eaa9 uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c Jwhs 1 2 NA NA NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
f74bad7734853d1d82554a53299bb301ae6e620a uuid:97193853-6939-4d7b-9e64-8aa784528360 Wueh 2 94 8 0 NA https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.G1.R1
knitr::kable(subtab2)
id submissions_id br009 odata_context
1e947c623494f58ac88260152d3a96b5eb54827d uuid:9f7a28fd-0e6c-4697-929d-013fd836419e Tg https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.R2
15d0e09d08698846c774c5370e0fa7cce443e2e9 uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 Heue https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.R2
ccf7b260ea1d10d3e734306b20ad45fcd154177e uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 Yeah https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.R2
5038e1d1a102d2e2cd9cf10a1e992b4c1f4ff27c uuid:09a89415-434a-4004-b611-188513e57b6b Yeay https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.R2
525a19d5bebeb47b4daf726f3a4010d0654b6377 uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c abc https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.R2
50d1efcda53e49ac9add84bb4a7ee860d645c577 uuid:97193853-6939-4d7b-9e64-8aa784528360 Hsu https://odk.idradata.org/v1/projects/1/forms/odk_training_lesson_BR.svc/$metadata#Submissions.R2

Exctracting it all to csv

This is using the RESTful api, not ODATA.
Will take all tables+subtables (i.e. repeats) and make a zipdfile you can download

# This command downloads a zipfile into my "subs" subdirectory
downloads_folder = "data_from_server"
submission_export(local_dir = fs::path(downloads_folder))
## data_from_server/odk_training_lesson_BR.zip
# This looks wierd but it's just naming the file to read and the folder to extract to
# (paste0 is just a concatenation function)
unzip(paste0(downloads_folder,"/", form_to_load, ".zip"), 
      exdir = paste0(downloads_folder,"/", form_to_load))

Now we can read each of the csv files one by one.
I didn’t bother getting the names of forms to load automatically, but there are ways to do it

main_from_zip <- read.csv2(paste0(downloads_folder,"/", form_to_load,"/", "odk_training_lesson_BR" , ".csv"), sep=",")
repeat1_from_zip <- read.csv2(paste0(downloads_folder,"/", form_to_load,"/", "odk_training_lesson_BR-R1" , ".csv"), sep=",")
repeat2_from_zip <- read.csv2(paste0(downloads_folder,"/", form_to_load,"/", "odk_training_lesson_BR-R2" , ".csv"), sep=",")

These files look a bit different, and variable names are not the same, but content is the same:

knitr::kable(main_from_zip)
SubmissionDate NOTE001 DM001 G1.R1_count BR003 BR005 BR006 BR007 BR008 R2_count meta.instanceID KEY SubmitterID SubmitterName AttachmentsPresent AttachmentsExpected Status
2020-04-25T04:03:51.216Z NA 2 2 0 3.0 NA 1 uuid:9f7a28fd-0e6c-4697-929d-013fd836419e uuid:9f7a28fd-0e6c-4697-929d-013fd836419e 19 Ed 0 0 NA
2020-04-15T03:36:58.486Z NA 1 1 1 3.0 NA 1 uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 9 Default Project User 0 0 NA
2020-04-14T01:31:46.302Z NA 3 3 1 956.0 NA 1 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 9 Default Project User 0 0 NA
2020-04-14T01:28:52.729Z NA 1 1 2 6568.0 NA 1 uuid:09a89415-434a-4004-b611-188513e57b6b uuid:09a89415-434a-4004-b611-188513e57b6b 9 Default Project User 0 0 NA
2020-04-14T01:28:52.222Z NA 1 1 1 32.0 NA 1 uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c 9 Default Project User 0 0 NA
2020-04-14T01:21:09.922Z NA 1 1 1 6446.0 NA 1 uuid:97193853-6939-4d7b-9e64-8aa784528360 uuid:97193853-6939-4d7b-9e64-8aa784528360 9 Default Project User 0 0 NA

Here’s one of the “repeats”.
With this method, id variables are called “KEY” and “PARENT_KEY” instead of “id” and “submission_id”.

knitr::kable(repeat1_from_zip)
DM002 DM003 DM004 DM005 BR001 BR002 PARENT_KEY KEY
R 1 5 NA NA uuid:9f7a28fd-0e6c-4697-929d-013fd836419e uuid:9f7a28fd-0e6c-4697-929d-013fd836419e/G1/R1[1]
T 2 2 NA NA uuid:9f7a28fd-0e6c-4697-929d-013fd836419e uuid:9f7a28fd-0e6c-4697-929d-013fd836419e/G1/R1[2]
Hsue 2 1 NA NA uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13 uuid:8e97fe68-0779-4ae9-9235-85e3ca299b13/G1/R1[1]
Person 1 2 3 NA NA uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37/G1/R1[1]
Person 2 2 1 NA NA uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37/G1/R1[2]
Person 3 2 60 6 1 26.0 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37 uuid:cf1b0896-3757-4460-9bff-1b006ad5cc37/G1/R1[3]
Hi devon 2 3 NA NA uuid:09a89415-434a-4004-b611-188513e57b6b uuid:09a89415-434a-4004-b611-188513e57b6b/G1/R1[1]
Jwhs 1 2 NA NA uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c uuid:ce050889-0b6c-45e8-aa6b-cca33061e53c/G1/R1[1]
Wueh 2 94 8 0 uuid:97193853-6939-4d7b-9e64-8aa784528360 uuid:97193853-6939-4d7b-9e64-8aa784528360/G1/R1[1]