This additional document shows the memory usage and accuracy metrics for each model built during the Data Science Capstone project. The most recent results show the tuning of the model. Earlier results show how I experimented with different approaches using a smaller data set.

Date Run time Model size Accuracy Top 5 Accuracy Comment
2016-01-22 1.49 mins 0.177 0.34 Interpolated model using 90% of data
2016-01-22 1.22 hours 179 MB 0.105 0.172 Quadgram using 90% of data
2016-01-22 1.62 hours 83.9 MB 0.148 0.283 Trigram using 90% of data
2016-01-22 21.8 mins 1.03 MB 0.122 0.268 Bigram using 90% of data
2016-01-21 3.33 mins 0.173 0.333 Retune with lambda.tri = 0.8 & lambda.quad = 0.7
2016-01-21 2.85 mins 0.173 0.334 Retune with lambda.tri = 0.8 & lambda.quad = 0.6
2016-01-21 2.7 mins 0.172 0.333 Retune with lambda.tri = 0.8 & lambda.quad = 0.4
2016-01-21 2.72 mins 0.16 0.321 Retune with lambda.tri = 0.8 & lambda.quad = 0
2016-01-21 2.76 mins 0.158 0.319 Retune with lambda.tri = 0.6 & lambda.quad = 0
2016-01-21 2.73 mins 0.159 0.32 Retune with lambda.tri = 0.7 & lambda.quad = 0
2016-01-21 2.96 mins 0.173 0.333 Check tuned Interpolated model lambda.trigram = 0.7
2016-01-21 2.82 mins 0.173 0.333 Interpolated model with 16000 Dictionary size
2016-01-21 38.6 mins 116 MB 0.095 0.155 Quadgram with 16000 Dictionary size
2016-01-20 30.2 mins 61.4 MB 0.143 0.272 Trigram with 16000 Dictionary size
2016-01-20 14 mins 1.03 MB 0.122 0.268 Bigram with 16000 Dictionary size
2016-01-20 3.26 mins 0.172 0.333 Interpolated model with 15000 Dictionary size
2016-01-20 44.6 mins 116 MB 0.096 0.156 Quadgram with 15000 Dictionary size
2016-01-20 30.2 mins 60.5 MB 0.143 0.273 Trigram with 15000 Dictionary size
2016-01-20 22.2 mins 965 kB 0.121 0.267 Bigram with 15000 Dictionary size
2016-01-20 3.03 mins 0.171 0.331 Interpolated model with 20000 Dictionary size
2016-01-20 2.99 mins 0.171 0.331 Interpolated model with 12000 Dictionary size
2016-01-20 31.4 mins 57 MB 0.143 0.275 Trigram with 12000 Dictionary size
2016-01-20 33.2 mins 115 MB 0.098 0.161 Quadgram with 12000 Dictionary size
2016-01-20 12.1 mins 772 kB 0.12 0.266 Bigram with 12000 Dictionary size
2016-01-19 2.45 mins 0.17 0.329 Interpolated model with 10000 Dictionary size
2016-01-19 1.07 hours 114 MB 0.1 0.165 Quadgram with 10000 Dictionary size
2016-01-19 25 mins 53.9 MB 0.143 0.276 Trigram with 10000 Dictionary size
2016-01-19 13.2 mins 643 kB 0.119 0.264 Bigram with 10000 Dictionary size
2016-01-18 1.99 mins 0.167 0.325 Interpolated model with 7000 Dictionary size
2016-01-18 38.7 mins 111 MB 0.104 0.176 Quadgram with 7000 Dictionary size
2016-01-18 23.3 mins 47.3 MB 0.143 0.279 Trigram with 7000 Dictionary size
2016-01-18 12.6 mins 451 kB 0.117 0.261 Bigram with 7000 Dictionary size
2016-01-17 2.17 mins 0.168 0.326 Rerun interpolated model without single letters
2016-01-17 33.1 mins 112 MB 0.101 0.17 Rerun quadgram model without single letters
2016-01-17 22.1 mins 49.8 MB 0.142 0.277 Rerun trigram model without single letters
2016-01-17 11.3 mins 515 kB 0.118 0.262 Rerun bigram model without single letters
2016-01-17 2.14 mins 0.169 0.329 Test with with lambda.tri = 0.8 & lambda.quad = 0.5
2016-01-17 2.74 mins 0.169 0.329 Test with with lambda.tri = 0.85 & lambda.quad = 0.5
2016-01-17 2.56 mins 0.169 0.329 Test with with lambda.tri = 0.85 & lambda.quad = 0.8
2016-01-17 2.61 mins 0.169 0.329 Test with with lambda.tri = 0.85 & lambda.quad = 0.6
2016-01-17 2.72 mins 0.169 0.328 Test with with lambda.tri = 0.85 & lambda.quad = 0.4
2016-01-17 2.06 mins 0.166 0.324 Test with with lambda.tri = 0.85 & lambda.quad = 0.2
2016-01-17 2.13 mins 0.156 0.316 Test with with lambda.tri = 0.85 & lambda.quad = 0
2016-01-17 2.24 mins 0.156 0.316 Test with with lambda.tri = 0.82 & lambda.quad = 0
2016-01-17 2.09 mins 0.156 0.316 Test with with lambda.tri = 0.9 & lambda.quad = 0
2016-01-17 2.45 mins 0.156 0.316 Test with with lambda.tri = 0.8 & lambda.quad = 0
2016-01-17 2.07 mins 0.154 0.313 Test with with lambda.tri = 0.6 & lambda.quad = 0
2016-01-17 2.3 mins 0.15 0.304 Test with with lambda.tri = 0.4 & lambda.quad = 0
2016-01-17 2.08 mins 0.139 0.287 Test with with lambda.tri = 0.2 & lambda.quad = 0
2016-01-17 2.04 mins 0.124 0.305 Test with with lambda.tri = 0 & lambda.quad = 1
2016-01-17 1.88 mins 0.149 0.316 Test with with lambda.tri = 1 & lambda.quad = 0
2016-01-17 2.62 mins 0.119 0.263 Interpolate with lambda.tri = 0 & lambda.quad = 0
2016-01-17 1.64 mins 0.119 0.263 Interpolate with lambda.tri = 0 & lambda.quad = 0
2016-01-16 2.58 mins 0.02 0.076 Interpolate with lambda.tri = 0 & lambda.quad = 0
2016-01-16 3.57 mins 0.021 0.076 Interpolated model with lambda.tri = 0.6 and lambda.quad = 0.3
2016-01-11 28.7 mins 113 MB 0.103 0.172 Quadgram with 8000 Dictionary size
2016-01-11 21.6 mins 50.6 MB 0.144 0.279 Trigram with 8000 Dictionary size
2016-01-11 10.8 mins 514 kB 0.119 0.263 Bigram with 8000 Dictionary size
2016-01-11 29.3 mins 115 MB 0.1 0.165 Quadgram with 10000 Dictionary size
2016-01-11 20.7 mins 54.7 MB 0.144 0.278 Trigram with 10000 Dictionary size
2016-01-11 10 mins 643 kB 0.12 0.265 Bigram with 10000 Dictionary size
2016-01-11 10.9 mins 643 kB Bigram with 10000 Dictionary size
2016-01-11 29.3 mins 118 MB 0.093 0.15 Quadgram with 20000 Dictionary size
2016-01-11 21.2 mins 65.1 MB 0.143 0.271 Trigram with 20000 Dictionary size
2016-01-11 10.1 mins 1.28 MB 0.123 0.27 Bigram with 20000 Dictionary size
2016-01-10 21.5 mins 69.1 MB 0.131 0.244 Trigram model with words in correct order
2016-01-10 22.7 mins 69.1 MB 0.011 Hashed trigram model with removed
2016-01-10 11 mins 1.91 MB 0.123 0.268 Hashed bigram model
2016-01-10 22.5 mins 71.6 MB 0.011 Hashed trigram model
2016-01-10 10.7 mins 1.91 MB 0.119 Hashed bigram model
2016-01-10 1.82 secs 46.5 kB Hashed trigram model
2016-01-10 1.79 secs 48.4 kB Hashed bigram model
2016-01-09 28.4 mins 121 MB 0.076 0.118 Quadgram model on training data
2016-01-08 54.5 secs 2.9 MB Hash all terms
2016-01-08 5.25 secs 138 kB Hash all terms
2016-01-08 53.2 secs 4.39 MB Benchmark Quadgram model
2016-01-08 3.96 secs 242 kB Benchmark Quadgram model
2016-01-08 52.2 secs 6.12 MB Benchmark on Carbon X1 Gen1
2016-01-08 4.58 secs 644 kB Benchmark on Carbon X1 Gen1
2016-01-08 0.484 secs 53.7 kB Benchmark on Carbon X1 Gen1
2016-01-02 1.04 mins 4.35 MB Quadgram model with different hash implementation
2016-01-02 6.77 mins 4.35 MB Hashed quadgram model
2016-01-02 37.6 secs 4.17 MB First quadgram model
2016-01-02 2.71 secs 231 kB First quadgram model
2016-01-02 5.65 mins 4.16 MB First bigram model
2016-01-02 20.3 secs 2.53 MB First bigram model
2016-01-02 2.81 secs 537 kB First bigram model
2016-01-02 0.55 secs 85.9 kB First bigram model
2016-01-02 14.6 mins 99.1 MB 0.143 Change join in trigram model, and measure perplexity
2016-01-02 40.6 secs 6.03 MB Change join in trigram model, and measure perplexity
2016-01-02 3.69 secs 627 kB Change join in trigram model, and measure perplexity
2016-01-02 0.588 secs 52.2 kB Change join in trigram model, and measure perplexity
2016-01-02 0.565 secs 52.2 kB Change join in trigram model, and measure perplexity
2016-01-02 0.603 secs 2.04 MB Remove unique trigrams, and measure perplexity
2016-01-02 40.8 secs 13.5 MB Include unique trigrams, and measure perplexity
2016-01-02 5.14 secs 2.05 MB Include unique trigrams, and measure perplexity
2015-12-31 14.5 mins 99.1 MB Restructured model with one row per bigram
2015-12-31 38.7 secs 6.15 MB Restructured model with one row per bigram
2015-12-31 3.58 secs 666 kB Restructured model with one row per bigram
2015-12-31 0.454 secs 65.7 kB Restructured model with one row per bigram
2015-12-31 8.41 mins 134 MB Have another go at the training model on the PC
2015-12-30 18.5 secs 5.13 MB Remove trigrams with word3 =
2015-12-30 1.82 secs 433 kB Remove trigrams with word3 =
2015-12-30 0.352 secs 42.4 kB Remove trigrams with word3 =
2015-12-30 18.2 secs 5.34 MB Replace infrequent words with
2015-12-30 2.07 secs 460 kB Replace infrequent words with
2015-12-30 0.444 secs 52.6 kB Replace infrequent words with
2015-12-30 19.2 secs 4.87 MB Before
2015-12-30 1.87 secs 390 kB Before
2015-12-30 0.295 secs 29.4 kB Before
2015-12-30 0.296 secs 52.6 kB Replace infrequent words with
2015-12-30 18.6 secs 4.87 MB Use data table for CountTrigrams
2015-12-30 1.65 secs 390 kB Use data table for CountTrigrams
2015-12-30 0.222 secs 29.4 kB Use data table for CountTrigrams
2015-12-30 1.06 secs 508 kB Baseline before Datatable changes
2015-12-28 1.83 hours 204 MB Rerun on Azure VM
2015-12-28 5.88 mins 13.3 MB Rerun on Azure VM
2015-12-27 1.3 mins 17.1 MB Using StringsAsFactors, but no index
2015-12-27 8.76 secs 2.69 MB Using StringsAsFactors, but no index
2015-12-27 3.01 secs 509 kB Using StringsAsFactors, but no index
2015-12-26 1.08 mins 11.6 MB Remove words occuring <6 from index
2015-12-26 5.84 secs 1.31 MB Remove words occuring <6 from index
2015-12-26 1.59 secs 170 kB Remove words occuring <6 from index
2015-12-26 7.22 secs 1.02 MB Remove unique words from index
2015-12-26 1.69 secs 108 kB Rerun after clearing memory
2015-12-26 2.65 secs 108 kB Remove unique words from index
2015-12-26 7.57 secs 884 kB Change tokens to integer
2015-12-26 2.04 secs 57.2 kB Change tokens to integer
2015-12-26 1.37 mins 13.4 MB Filter out unique trigrams
2015-12-26 1.28 mins 171 MB Larger dataset
2015-12-26 8.9 secs 24.7 MB Trigrams split into 3 word vectors
2015-12-26 1.37 secs 3.44 MB Trigrams split into 3 word vectors
2015-12-26 24.4 secs 29.2 MB Dev2
2015-12-26 2.53 secs 4.11 MB First model