The First Example

Welcome to Desktop Pro 1.2

RStudio Desktop Pro

  • New in 1.2
  • Requires a license

Let’s test desktop licensing

Test connecting to a license server

  • Testing included
    1. Make sure correct port works
    2. Make sure incorrect port fails gracefully

Everything passed—great! All done.

Ooh, ooh, how about this?

…Then I felt like entering a negative number.

Negative numbers outcome

  • Valid ports values go up to 65535 (2^16 - 1)

  • Negative numbers wrap around like a dial1

But negative numbers aren’t valid!

Why would you enter a negative number? 

But negative numbers aren’t valid!

Why would you enter a negative number?

Because you can.

Negative Numbers Aren’t Valid!

Why would you enter a negative number?

Because you can.

\(\Rightarrow\) Need to be aware of & handle improbable cases so that RStudio doesn’t break.

Manual. Exploratory. Black Box.

This was an example of
manual exploratory black box testing.

Manual. Exploratory. Black Box.

This was an example of
manual exploratory black box testing.

  • Manual = Done by a person

Manual. Exploratory. Black Box.

This was an example of
manual exploratory black box testing.

  • Manual = Done by a person
  • Exploratory = “Ooh, what does this do?”

Manual. Exploratory. Black Box.

This was an example of
manual exploratory black box testing.

  • Manual = Done by a person
  • Exploratory = “Ooh, what does this do?”
  • Black box = Like a typical user—functionality only, without exposure to the underlying code

Manual. Exploratory. Black Box.

Compare to
exploratory Scripted = detailed test cases based on specs.
 
Or, perhaps
manual Automated = predefined & run automatically (!)

Channeling the User

The Expert User

Stumbles across strange and interesting problems

The Novice User

Stumbles across strange and interesting problems

Gotta try things we don’t expect

Some more Desktop Pro license testing

  • Straightforward connectivity testing: what happens when
    1. License server is down
    2. Server is blocking requests
    3. Network is unreachable
  • Done by configuring RStudio, server, etc.

Gotta try things we don’t expect

 
 
 

…and unplugging the cable from the wall.

Perfect timing

 
Testing

  • using RStudio to create a PowerPoint presentation with R Markdown

Perfect timing

 
Testing

  • using RStudio to create a PowerPoint presentation with R Markdown

by

  • using RStudio to create a PowerPoint presentation with R Markdown for Show and Tell.

Perfect timing

Testing

  • using RStudio to create a PowerPoint presentation with R Markdown

by

  • using RStudio to create a PowerPoint presentation with R Markdown for Show and Tell.

i.e. dogfooding

What the…?

Then suddenly, while poking around,
files stopped opening on Ubuntu.
Sometimes.
 

  • Can’t just file a bug that says “files don’t open”
  • Need to identify failure vs. success conditions

Investigation components

RStudio Server
Files Pane
Mac
1.2.695
Open File
Windows
1.2.679

Ubuntu
Red Hat
1.2.707
SUSE
1.1.453
RStudio Desktop

Test walkthrough

Behavioral issues

Behavior was complicated by:

Behavioral issues

Behavior was complicated by:

  1. Browser caching

Behavioral issues

Behavior was complicated by:

  1. Browser caching
  2. A resolved issue

Behavioral issues

Behavior was complicated by:

  1. Browser caching
  2. A resolved issue
  3. Newer versions on top of older versions

Behavioral issues

Behavior was complicated by:

  1. Browser caching
  2. A resolved issue
  3. Newer versions on top of older versions
    \(\Rightarrow\) A reminder that users upgrade, so we need to test upgrades vs. fresh installations
     

Behavioral issues

Behavior was complicated by:

  1. Browser caching
  2. A resolved issue
  3. Newer versions on top of older versions
    \(\Rightarrow\) A reminder that users upgrade, so we need to test upgrades vs. fresh installations
     

Once the cause & resolution were identified, validation was a little easier

Validation

The arrows lined up

Last slide

  • In a sense, y’all are doing this by using our products

Last slide

  • In a sense, y’all are doing this by using our products
  • Send “Hey, something weird is going on” over QA way—it’s what we do

Last slide

  • In a sense, y’all are doing this by using our products
  • Send “Hey, something weird is going on” over QA way—it’s what we do
  • Plus, this kind of detective of work is fun!