Project Description

Almost every web service you join will require you to come up with a password. But what makes a good password? In June 2017 the the National Institute of Standards and Technology (NIST) published publication 800-63B titled Digital Identity Guidelines: Authentication and Lifecycle Management. This publication doesn’t tell you what is a good password, but it does have specific rules for what is a bad password.

In this project, you will take a list of user passwords and, using publication 800-63B, you will write code that automatically detects and flags the bad passwords.

To complete this project, you need to know how to manipulate text using the stringr package and be familiar with regular expressions.

1 The NIST Special Publication 800-63B

If you – 50 years ago – needed to come up with a secret password you were probably part of a secret espionage organization or (more likely) you were pretending to be a spy when playing as a kid. Today, many of us are forced to come up with new passwords all the time when signing into sites and apps. As a password inventeur it is your responsibility to come up with good, hard-to-crack passwords. But it is also in the interest of sites and apps to make sure that you use good passwords. The problem is that it’s really hard to define what makes a good password. However, the National Institute of Standards and Technology (NIST) knows what the second best thing is: To make sure you’re at least not using a bad password.

In this notebook, we will go through the rules in NIST Special Publication 800-63B which details what checks a verifier (what the NIST calls a second party responsible for storing and verifying passwords) should perform to make sure users don’t pick bad passwords. We will go through the passwords of users from a fictional company and use R to flag the users with bad passwords. But us being able to do this already means the fictional company is breaking one of the rules of 800-63B:

Verifiers SHALL store memorized secrets in a form that is resistant to offline attacks. Memorized secrets SHALL be salted and hashed using a suitable one-way key derivation function.

That is, never save users’ passwords in plaintext, always encrypt the passwords! Keeping this in mind for the next time we’re building a password management system, let’s load in the data.

Warning: The list of passwords and the fictional user database both contain real passwords leaked from real websites. These passwords have not been filtered in any way and include words that are explicit, derogatory and offensive.

## [1] 982
## # A tibble: 12 x 3
##       id user_name        password        
##    <int> <chr>            <chr>           
##  1     1 vance.jennings   joobheco        
##  2     2 consuelo.eaton   0869347314      
##  3     3 mitchel.perkins  fabypotter      
##  4     4 odessa.vaughan   aharney88       
##  5     5 araceli.wilder   acecdn3000      
##  6     6 shawn.harrington 5278049         
##  7     7 evelyn.gay       master          
##  8     8 noreen.hale      murphy          
##  9     9 gladys.ward      lwsves2         
## 10    10 brant.zimmerman  1190KAREN5572497
## 11    11 leanna.abbott    aivlys24        
## 12    12 milford.hubbard  hubbard

2 Passwords should not be too short

If we take a look at the first 12 users above we already see some bad passwords. But let’s not get ahead of ourselves and start flagging passwords manually. What is the first thing we should check according to the NIST Special Publication 800-63B?

Verifiers SHALL require subscriber-chosen memorized secrets to be at least 8 characters in length.

Ok, so the passwords of our users shouldn’t be too short. Let’s start by checking that!

## [1] 376
## # A tibble: 6 x 5
##      id user_name        password   length too_short
##   <int> <chr>            <chr>       <int> <lgl>    
## 1     1 vance.jennings   joobheco        8 F        
## 2     2 consuelo.eaton   0869347314     10 F        
## 3     3 mitchel.perkins  fabypotter     10 F        
## 4     4 odessa.vaughan   aharney88       9 F        
## 5     5 araceli.wilder   acecdn3000     10 F        
## 6     6 shawn.harrington 5278049         7 T

3 Common passwords people use

Already this simple rule flagged a couple of offenders among the first 12 users. Next up in Special Publication 800-63B is the rule that

verifiers SHALL compare the prospective secrets against a list that contains values known to be commonly-used, expected, or compromised.

  • Passwords obtained from previous breach corpuses.
  • Dictionary words.
  • Repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).
  • Context-specific words, such as the name of the service, the username, and derivatives thereof.

We’re going to check these in order and start with Passwords obtained from previous breach corpuses, that is, websites where hackers have leaked all the users’ passwords. As many websites don’t follow the NIST guidelines and encrypt passwords there now exist large lists of the most popular passwords. Let’s start by loading in the 10,000 most common passwords which I’ve taken from here.

##   [1] "123456"     "password"   "12345678"   "qwerty"     "123456789" 
##   [6] "12345"      "1234"       "111111"     "1234567"    "dragon"    
##  [11] "123123"     "baseball"   "abc123"     "football"   "monkey"    
##  [16] "letmein"    "696969"     "shadow"     "master"     "666666"    
##  [21] "qwertyuiop" "123321"     "mustang"    "1234567890" "michael"   
##  [26] "654321"     "pussy"      "superman"   "1qaz2wsx"   "7777777"   
##  [31] "fuckyou"    "121212"     "000000"     "qazwsx"     "123qwe"    
##  [36] "killer"     "trustno1"   "jordan"     "jennifer"   "zxcvbnm"   
##  [41] "asdfgh"     "hunter"     "buster"     "soccer"     "harley"    
##  [46] "batman"     "andrew"     "tigger"     "sunshine"   "iloveyou"  
##  [51] "fuckme"     "2000"       "charlie"    "robert"     "thomas"    
##  [56] "hockey"     "ranger"     "daniel"     "starwars"   "klaster"   
##  [61] "112233"     "george"     "asshole"    "computer"   "michelle"  
##  [66] "jessica"    "pepper"     "1111"       "zxcvbn"     "555555"    
##  [71] "11111111"   "131313"     "freedom"    "777777"     "pass"      
##  [76] "fuck"       "maggie"     "159753"     "aaaaaa"     "ginger"    
##  [81] "princess"   "joshua"     "cheese"     "amanda"     "summer"    
##  [86] "love"       "ashley"     "6969"       "nicole"     "chelsea"   
##  [91] "biteme"     "matthew"    "access"     "yankees"    "987654321" 
##  [96] "dallas"     "austin"     "thunder"    "taylor"     "matrix"

4 Passwords should not be common passwords

The list of passwords was ordered, with the most common passwords first, and so we shouldn’t be surprised to see passwords like 123456 and qwerty above. As hackers also have access to this list of common passwords, it’s important that none of our users use these passwords!

Let’s flag all the passwords in our user database that are among the top 10,000 used passwords.

## [1] 129
## # A tibble: 12 x 6
##       id user_name        password         length too_short common_passwo…
##    <int> <chr>            <chr>             <int> <lgl>     <lgl>         
##  1     1 vance.jennings   joobheco              8 F         F             
##  2     2 consuelo.eaton   0869347314           10 F         F             
##  3     3 mitchel.perkins  fabypotter           10 F         F             
##  4     4 odessa.vaughan   aharney88             9 F         F             
##  5     5 araceli.wilder   acecdn3000           10 F         F             
##  6     6 shawn.harrington 5278049               7 T         F             
##  7     7 evelyn.gay       master                6 T         T             
##  8     8 noreen.hale      murphy                6 T         T             
##  9     9 gladys.ward      lwsves2               7 T         F             
## 10    10 brant.zimmerman  1190KAREN5572497     16 F         F             
## 11    11 leanna.abbott    aivlys24              8 F         F             
## 12    12 milford.hubbard  hubbard               7 T         F

5 Passwords should not be common words

It turns out many of our passwords were common English words too! Next up on the NIST list:

Verifiers SHALL compare the prospective secrets against a list that contains […] context-specific words, such as the name of the service, the username, and derivatives thereof.

Ok, so there are many things we could check here. One thing to notice is that our users’ usernames consist of their first names and last names separated by a dot. For now, let’s just flag passwords that are the same as either a user’s first or last name.

## [1] 137
## # A tibble: 12 x 7
##       id user_name        password         length too_short common… commo…
##    <int> <chr>            <chr>             <int> <lgl>     <lgl>   <lgl> 
##  1     1 vance.jennings   joobheco              8 F         F       F     
##  2     2 consuelo.eaton   0869347314           10 F         F       F     
##  3     3 mitchel.perkins  fabypotter           10 F         F       F     
##  4     4 odessa.vaughan   aharney88             9 F         F       F     
##  5     5 araceli.wilder   acecdn3000           10 F         F       F     
##  6     6 shawn.harrington 5278049               7 T         F       F     
##  7     7 evelyn.gay       master                6 T         T       T     
##  8     8 noreen.hale      murphy                6 T         T       T     
##  9     9 gladys.ward      lwsves2               7 T         F       F     
## 10    10 brant.zimmerman  1190KAREN5572497     16 F         F       F     
## 11    11 leanna.abbott    aivlys24              8 F         F       F     
## 12    12 milford.hubbard  hubbard               7 T         F       F

6 Passwords should not be your name

It turns out many of our passwords were common English words too! Next up on the NIST list:

Verifiers SHALL compare the prospective secrets against a list that contains […] context-specific words, such as the name of the service, the username, and derivatives thereof.

Ok, so there are many things we could check here. One thing to notice is that our users’ usernames consist of their first names and last names separated by a dot. For now, let’s just flag passwords that are the same as either a user’s first or last name.

## [1] 50
## # A tibble: 12 x 10
##       id user_name        passw… leng… too_… comm… comm… firs… last… uses…
##    <int> <chr>            <chr>  <int> <lgl> <lgl> <lgl> <chr> <chr> <lgl>
##  1     1 vance.jennings   joobh…     8 F     F     F     vance jenn… F    
##  2     2 consuelo.eaton   08693…    10 F     F     F     cons… eaton F    
##  3     3 mitchel.perkins  fabyp…    10 F     F     F     mitc… perk… F    
##  4     4 odessa.vaughan   aharn…     9 F     F     F     odes… vaug… F    
##  5     5 araceli.wilder   acecd…    10 F     F     F     arac… wild… F    
##  6     6 shawn.harrington 52780…     7 T     F     F     shawn harr… F    
##  7     7 evelyn.gay       master     6 T     T     T     evel… gay   F    
##  8     8 noreen.hale      murphy     6 T     T     T     nore… hale  F    
##  9     9 gladys.ward      lwsve…     7 T     F     F     glad… ward  F    
## 10    10 brant.zimmerman  1190K…    16 F     F     F     brant zimm… F    
## 11    11 leanna.abbott    aivly…     8 F     F     F     lean… abbo… F    
## 12    12 milford.hubbard  hubba…     7 T     F     F     milf… hubb… T

7 Passwords should not be repetitive

Milford Hubbard (user number 12 above), what where you thinking!? Ok, so the last thing we are going to check is a bit tricky:

verifiers SHALL compare the prospective secrets [so that they don’t contain] repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).

  1. Passwords should not be repetitive Milford Hubbard (user number 12 above), what where you thinking!? Ok, so the last thing we are going to check is a bit tricky:

verifiers SHALL compare the prospective secrets [so that they don’t contain] repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).

This is tricky to check because what is repetitive is hard to define. Is 11111 repetitive? Yes! Is 12345 repetitive? Well, kind of. Is 13579 repetitive? Maybe not..? To check for repetitiveness can be arbitrarily complex, but here we’re only going to do something simple. We’re going to flag all passwords that contain 4 or more repeated characters.

## # A tibble: 6 x 12
##      id user_… pass… leng… too_… comm… comm… firs… last… uses… max_… too_…
##   <int> <chr>  <chr> <int> <lgl> <lgl> <lgl> <chr> <chr> <lgl> <int> <lgl>
## 1   147 patti… 5555…     6 T     T     F     patti dixon F         6 T    
## 2   573 corne… 5555…     6 T     T     F     corn… brad… F         6 T    
## 3   645 essie… 11111     5 T     T     F     essie lopez F         5 T    
## 4   799 charl… 8888…     6 T     T     F     char… key   F         6 T    
## 5   808 thurm… rinn…     8 F     F     F     thur… osbo… F         4 T    
## 6   942 mitch… aaaa…     6 T     T     F     mitch ferg… F         6 T

8 All together now!

Now we have implemented all the basic tests for bad passwords suggested by NIST Special Publication 800-63B! What’s left is just to flag all bad passwords and maybe to send these users an e-mail that strongly suggests they change their password.

## [1] 424
##   [1] FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE FALSE FALSE
##  [12]  TRUE FALSE  TRUE FALSE  TRUE  TRUE  TRUE  TRUE FALSE FALSE  TRUE
##  [23]  TRUE FALSE FALSE  TRUE  TRUE FALSE  TRUE FALSE  TRUE FALSE FALSE
##  [34] FALSE  TRUE FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE  TRUE
##  [45] FALSE  TRUE FALSE  TRUE  TRUE FALSE FALSE  TRUE  TRUE FALSE FALSE
##  [56] FALSE  TRUE FALSE  TRUE  TRUE FALSE FALSE  TRUE  TRUE  TRUE FALSE
##  [67] FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
##  [78] FALSE  TRUE FALSE FALSE  TRUE FALSE  TRUE  TRUE FALSE FALSE  TRUE
##  [89] FALSE  TRUE FALSE  TRUE  TRUE FALSE FALSE  TRUE FALSE  TRUE FALSE
## [100] FALSE

9 Otherwise, the password should be up to the user

In this notebook, we’ve implemented the password checks recommended by the NIST Special Publication 800-63B. It’s certainly possible to better implement these checks, for example, by using a longer list of common passwords. Also note that the NIST checks in no way guarantee that a chosen password is good, just that it’s not obviously bad.

Apart from the checks we’ve implemented above the NIST is also clear with what password rules should not be imposed:

Verifiers SHOULD NOT impose other composition rules (e.g., requiring mixtures of different character types or prohibiting consecutively repeated characters) for memorized secrets. Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically).

So the next time a website or app tells you to “include both a number, symbol and an upper and lower case character in your password” you should send them a copy of NIST Special Publication 800-63B.