goveda: a package for use and consuption of open data

https://www.datos.gov.co/

Service plataforms enables customers and users to download the data via a huge number of open standards (CSV, JSON, etc.) so there is no risk of lock-in at the data level.

• Socrata is a software-as-a-service platform that provides a cloud-based solution for open data publishing and visualization

• CKAN is an open source project, developed by the Open Knowledge Foundation, that lets users provision open data catalogs and, in some cases, visualizations and APIs.

• Succesuful cases: Colombia open data porta, Mexico open data portal, Paraguay Open Data portal, etc.

• get_socrata_metada: fetch a detailed metada of open datasets (extract json and basic list of datasets available from a Socrata domain

• search_data: Shows a list object with with available datasets by keywords/tags. The list contains four things:

• eda_opendata: Generate a basic report in rmarkdown or shiny with selected datasets from an open data portal.

Future: a basic recommendation system.

Description fetch a detailed metada of open datasets (extract json and basic list of datasets available from a Socrata domain

Usage

get_socrata_metada(url)

Arguments

• url - A Socrata URL. This simply points to the site root.

Details a R data frame containing a listing of datasets along with detailed metadata. Next field are preserved for every open dataset:

• id, name, attribution, averageRating, category, createdAt, description, displayType, downloadCount, hideFromCatalog, hideFromDataJson
• indexUpdatedAt, licenseId, newBackend, numberOfComments, oid, provenance, publicationAppendEnabled, publicationDate, publicationGroup
• publicationStage, rowsUpdatedAt, rowsUpdatedBy, tableId, totalTimesRated, viewCount, viewLastModified, viewType, columns, grants, license, metadata
• owner, query, rights, tableAuthor, tags, flags, attributionLink, blobFilename, blobFileSize, blobId, blobMimeType, rowClass, previewImageId, * childViews, displayFormat, iconUrl, rowIdentifierColumnId, ratings disabledFeatureFlags, accessLevel, landingPage, issued, @type, modified
• keyword, contactPoint, publisher, identifier, title, distribution, theme

Value The function returns a dataframe with detailed information about datasets.

Description Shows a list object with with available datasets by keywords/tags. The list contains four things:

• tabular datasets: a dataframe with four columns (dataset name, attribution (public agency), number of rows and columns size of dataset (Mb) and elapsed time for R export.
• geo: a dataframe with general information about the geo file.
• href: a dataframe with general information about the href file.
• blobby: a dataframe with general information about the href file.

Usage

search_data(metadata, tags)

Arguments

• metadata - a dataframe get from get_metadata function
  
• tags a character vector with keywords and/or tags required for searching in dataset name, tag field or public agency field

Details The selected sample is drawn according to a selection-rejection (list-sequential) algorithm

Value The function returns a list of 4 dataframes. The 4 dataframes contain information of tabular, geo, href and blobby data of open data portal.

• A output from a tabular data (searching Bogotá datasets):
• > search_data(metadata = metada_colombia, tags = “Bogotá”)[[1]]

Description Generate a basic report in rmarkdown or shiny with selected datasets from an open data portal.

Usage

eda_opendata(metadata, ids, tags)

Arguments

• metadata - a dataframe get from get_metadata function
• ids - a character vector with datasets id required. if ids and tags arguments are NULL all datasets are analysed
  
• tags - a character vector with tags/keywords required for analysis. if ids and tags arguments are NULL all datasets are analysed.

Details The selected sample is drawn according to a selection-rejection (list-sequential) algorithm

Value The function returns a list of 4 dataframes. The 4 dataframes contain information of tabular, geo, href and blobby data of open data portal.

• Most datasets to (2017-03-04) are of tabular type.
• A frecuency analysis of data type in colombia open is show below:

• Every day is added new open data to “datos abiertos” portal:

• Some common topics in Colombia open data are:

• Distribution of number of views of Colombian datasets:

• A descriptive statistics summary of view counts is presented below: