Signatories

Project team

gwynn sturdevant, Jonathan Godfrey, Kelli Johnson, and Andrea Blasco

Contributors

Liz Hare

Consulted

Steven Randazzo, Barret Schloerke, and Gary Ritchie

The Problem

In the era of big data and digital transformation, a fundamental societal goal is providing blind and low vision (BLV) people with accessibility to websites and data visualizations.

In the U.S. alone, about 93 million Americans have or are at near term high risk of severe vision loss; and global trends suggest the number of visually impaired people is rising due to aging and other known risk factors. These people face daily challenges when interacting with online information services, such as those offered by the government, and are at risk of being excluded from many of the benefits of a digital society.

The COVID-19 pandemic provides perhaps the best illustration of this problem. While communication about the pandemic relied heavily on graphs and data visualizations to break down key public-health messages to people. This communication ended up benefiting only the sighted ones, with key concepts of the pandemic, such as the “Flattening the Curve” plot, being explained to the public in a way wholly inaccessible to BLV people.

In the context of the COVID-19 pandemic, researchers eventually developed solutions to address the accessibility problem, such as a “sonified” — non-speech audio used to represent information or data — of the flatten the curve plot. However, this remains a non-systematic solution.

Even the guidelines for communication to people with disabilities, such as those provided by the UN Convention on the Rights of Persons with Disabilities and Web Accessibility Initiative, appear insufficient and new tools must be developed to make it easy for governments and other organizations to follow these requirements.

The main objective of this project is to explore ways to communicate information contained in graphs and data visualizations to people with low vision or blindness. Our specific goal is to develop a new algorithm that helps address one critical aspect of the problem: the automatic generation of text alternatives (alt-text) to graph and data visualizations in the communication of science.

Alt-text is a common feature of data visualizations that includes the type of chart, the type of data, and the reason for including the chart. For Figure 1, appropriate alt-text is “Bar chart of gun murders per 100,000 people where America’s murder rate is 6 times worse than Canada, and 30 times Australia.”
Figure 1

Indeed, automated text is more effective and practical than training thousands of people how to write alt-text effectively. However, developing a comprehensive algorithm for automated alt-text can be challenging. More specifically, we will build upon a widely used library, ggplot, that includes sophisticated functions to generate scientific graphs using the R programming language. Currently, alt-text for ggplot produce texts that are less informative than needed. This project aims to improve the state-of-the-art alt-text generation algorithms by making the outcome both informative and succinct. Although repeating the caption is a good start, we can do better as some authors do not write appropriate or meaningful captions (captions are often too long or incomplete for effective alt-text representation).

Generating better alt-text for graphs and data visualizations will benefit not only BLV people but also thousands of organizations and their workers that must produce accessible documents. The default html from RMarkdown is used extensively throughout multiple organizations and complies with all Web Consortium Accessibility Guidelines except when graphics are involved. Because of this, many organizations are being forced to change their workflows or abandon RMarkdown altogether. We want to avoid people turning away from RMarkdown because Web Accessiblity Initiative guidelines are not currently met by automating and improving the alt-text.

The proposal

Overview

This project aims to use crowd sourcing on social media and Amazon Mechincal Turk to create high-quality alt-text for data visualizations that benefit BLV people and others who need to meet accessibility requirements for documents.

Detail

  1. Description of the problem - our goal is to make ggplot more user-friendly for BLV people. Currently alt-text for ggplot is verbose and focuses too much on axes and labels. MiR [Minorities in R] project has a grant to work with Tidy Tuesday to teach the RStats social media communities about alt-text. This project is an extension of the MiR project that improves the quality of alt-text and crowd-sources accessibility for ggplot to the RStats social media communities.
    1. A ggplot is a list that tells R how to make a plot
    2. We plan to include another element to this list that includes alt-text
    3. If BrailleR is loaded ggplot interacts with a screen reader and speaks alt-text
    4. The purpose of this project is to develop the alt-text for the screen reader by using crowdsourcing on social media and Amazon Mechanical Turk.
    5. Plots in the same order as the R for graphics cookbook
    6. Screen-reader video of a ggplot graph so that our audience can contrast after the project to see improvement
  2. Engagement plan
    1. People to post on social media about it:
      1. Hadley
      2. Mara
      3. Thomas Lumley
      4. Andrew Gelman
      5. Tom Mock
      6. RForwards
      7. RConsortium
      8. Others
    2. ggplot2 mailing list.
    3. Give away ipads to a random sample of people who have provided us with alt-text
    4. Develop a logo
    5. Print logo on t-shirts, mugs, and stickers and distribute them
    6. Time-bound
    7. Immediate reward - when time closes, select those that are eligible for the reward and randomly draw and ship out ipads
    8. Use Amazon Mechanical Turk
  3. Eligibility criteria for ipad
    1. High-quality solutions that align with MiR project
    2. Have the appropriate hashtag so we can find it
      1. Hashtag - One for each plot?
      2. Hashtag - One for the whole time-bound period?
    3. Participate in each one?
  4. Output - will need to analyze texts to come to a consensus
    1. Show the participants how it worked and helped.
    2. Video of better screen reader?
    3. BlindRUG comments about how much better things are?
    4. Begin to understand what we need to automagically create alt-text for a ggplot object.

Project plan

Start-up phase

Weeks 1:5;

  • Invite new team members via Twitter/LinkedIn
  • Team orientation for all members
  • Code of conduct and a website

We expect the project team of this grant to be the main drivers of this research. We will also need an active community that is willing to help.

Technical delivery

Weeks 6 - end;

  • Community engagement:
    • Regularly publish plots on social media to crowd source development of alt-text
      • write post
      • find graphic, and store ggplot list for algorithmic development
      • post on social media
      • publicize post
    • Place post on Amazon Mechanical Turk
  • Data collection:
    • Scrape alt-text
      • clean data
      • data storage
      • rank for criteria such as clarity and informativeness
    • Coalesce alt-text into one brief statement
      • use NLP to find common words
      • look for differences in alt-text with high rankings for different criteria
    • Use data to begin creating algorithm for alt-text automation

Other aspects

We will have multiple posts on social media to increase engagement, and ask leaders in the R social media community to post our work to encourage participation. Hadley, Mara, Andrew Gelman, Tom Mock, RForwards, RConsortium, and others may help with this.

Requirements

People

gwynn, Jonathan, Kelli, Andrea, and others who volunteer will all work to harness social media to create alt-text for ggplot.

Processes

We will follow the R Consortium Code of Conduct. We anticipate having a meeting with the Infrastructure Steering Committee after 5 weeks, then quarterly updates.

Tools & Tech

To ensure engagement we will to print special edition t-shirts, stickers, and mugs with a vignette explaining the importance of alt-text in figures. We plan to distribute these widely at conferences this July and August and through the mail. We will also have a random draw for several ipads for those who meet eligibility criteria and use Amazon Mechanical Turk who will surely participate.

Funding

We anticipate the costs of this project will be in purchasing ipads, developing and disseminating a logo, building a website, Amazon Mechincal Turk, scraping alt-text from social media, and coalescing the alt-text.

Item Cost (USD)
Milestone 1: Building website $2,000
Milestone 2: Scraping and coalescing alt-text $5,000
Milestone 3: 2 Ipads $1,500
Milestone 4: Logo development $500
Milestone 5: Prints and distribution of t-shirts, stickers, and mugs $3,000
Milestone 6: Amazon Mechanical Turk $2,000
Milestone 7: Code development for alt-text algorithm $5,000

Summary

To incentivize quality participation we have included a raffle for those that meet the requirements and t-shirts, stickers, and mugs with our logo. The most difficult part of open innovation is engagement, to address this raffles and prizes are terrific motivators. The use of Amazon Mechanical Turk also ensures our success.

Success

Definition of done

We define success as beginning to understand how to algorithmically develop alt-text for ggplot that is useful for BLV people and others that need accessible documents. The current expectation is that hundreds of thousands of developers learn to write suitable alt-text, something that we believe is not feasible and can be done more quickly using technology. Our goal is to move closer to creating an algorithm that automates and builds alt-text using the information in a ggplot object. Collating crowd-sourced alt-text from social media is a crucial part of this process, as it provides us data with which to build the algorithm.

Measuring success

Moving closer to improved alt-text in ggplot will show success.

Future work

Extensions of this work include connecting to data through nonvisual senses through R. As Leland Wilkinson said “The graphs we have been talking about cannot be seen, heard, tasted, or otherwise perceived. It is the job of the Aesthetic object to act as a function on the elements of a CGraph [coordinate graph] and translate them into real numbers or characters that control some display device, such as a CRT [cathoid ray tube], plotter, sound generator, or even odor generator. Once we have done this, we have what I call a graphic. A graphic is a perceivable graph.” There are many technological feasible ways that we can interact with computers beyond that of a mouse, keyboard, and screen which will improve the accessibility of data for BLV people and improve the workflow for all. We call this process data vivification

Key risks

The big concern with open innovation is a lack of engagement from the community. The R social media community is active and we anticipate lots of engagement, the Ipad raffle and distribution of logo will also encourage participation. Additionally, the R community cares deeply for its members. We are also leveraging this risk by using Amazon Mechanical Turk, where we are certain to find participants.