Compiler
Adam Chandler
Mon Dec 8 11:32:34 2014
Purpose:
1. create a baseline for Voyager load performance
2. identify the characteristics of the slowest jobs
Data source:
This analysis builds on work Pete did in the fall to create a log of job load performance. The source data is : http://lstools.library.cornell.edu/mysql/bulkimplog.cgi
I added two fields to facilite analysis and visualization of the data:
1. Secs.Bib = round(Durationsecs / Records, 2))
2. Durationsecs = Duration converted to seconds
471 Jobs in Sample
## Classes 'tbl_df', 'tbl' and 'data.frame': 471 obs. of 14 variables:
## $ Start : chr "11/1/2014 7:35" "11/1/2014 16:55" "11/2/2014 6:05" "11/2/2014 10:27" ...
## $ WeekDay : chr "Sat" "Sat" "Sun" "Sun" ...
## $ Records : int 4240 13329 1 183 29 82715 220 9153 166 1 ...
## $ Duration : chr "0:50:53" "2:23:04" "0:00:13" "0:04:56" ...
## $ Bibs.Sec : num 1.38 1.55 0.07 0.61 0.48 4.5 0.77 4.48 2.15 0.09 ...
## $ BulkImpRule : chr "BIBUPD" "BIBMAINT" "BIBUPD" "MITCN" ...
## $ Netid : chr "lbatch" "gcb2" "lbatch" "nv19" ...
## $ KeywordIndexing: chr "no" "no" "yes" "yes" ...
## $ Added : int 0 0 0 183 29 0 0 5502 0 1 ...
## $ Merged : int 0 0 0 0 0 0 0 3527 166 0 ...
## $ Replaced : int 4240 13329 1 0 0 82715 220 0 0 0 ...
## $ Filename : chr "lbatch.oclc_mapping.BIBUPD.oclc_mapping.mrc" "gcb2..BIBMAINT.sersol948f_reload.mrc" "lbatch.oclc_mapping.BIBUPD.oclc_mapping.mrc" "nv19..MITCN.CLMV_Supplement3_M8DONE.mrc" ...
## $ Durationsecs : num 3053 8584 13 296 60 ...
## $ Secs.Bib : num 0.72 0.64 13 1.62 2.07 0.22 1.3 0.22 0.46 11 ...
## Source: local data frame [40 x 5]
##
## BulkImpRule totalrecords totaljobs totaldurationsecs Secs.Bib
## 1 BIBMAINT 1658014 86 496873 0.30
## 2 BIBUPD 57466 29 20204 0.35
## 3 SERSOLUN 50297 2 14473 0.29
## 4 AUTHMERG 43688 8 8954 0.20
## 5 SERSOL 30730 4 6316 0.21
## 6 BIBSUP 5946 12 2284 0.38
## 7 SERSOLD 2617 2 1777 0.68
## 8 MARCADIA 1009 2 1825 1.81
## 9 YANKEE 899 3 1490 1.66
## 10 CAMBEIRO 852 21 1724 2.02
## 11 MITCN 836 8 1639 1.96
## 12 MARCIVE 547 1 929 1.70
## 13 NETLIB2 511 3 969 1.90
## 14 NETLIB1 286 2 434 1.52
## 15 YANKEEFO 278 35 2713 9.76
## 16 HARRFULL 270 40 661 2.45
## 17 COUTUKFO 242 35 2114 8.74
## 18 AMBASFO 194 24 1492 7.69
## 19 HARRASFO 174 25 1862 10.70
## 20 ELECCON 102 15 581 5.70
## 21 CASPLUS 85 30 409 4.81
## 22 CASALINI 76 16 488 6.42
## 23 ERASMUS 53 12 520 9.81
## 24 BENFO 35 7 539 15.40
## 25 DKAFO 34 8 438 12.88
## 26 AUXFO 30 13 388 12.93
## 27 PANMUNFO 21 3 285 13.57
## 28 HARRASS 19 2 34 1.79
## 29 AMAPDBFO 6 6 24 4.00
## 30 KINOFO 6 2 178 29.67
## 31 COUPDBFO 3 3 8 2.67
## 32 HARRASSO 3 1 9 3.00
## 33 BACHFO 2 2 12 6.00
## 34 ISISFO 2 2 24 12.00
## 35 VANSCHFO 2 2 18 9.00
## 36 AKAT 1 1 6 6.00
## 37 CAMBEFO 1 1 4 4.00
## 38 CENTROFO 1 1 15 15.00
## 39 CNPIECFO 1 1 5 5.00
## 40 SOFIAFO 1 1 17 17.00
## [1] "# of jobs: 41"
## [1] "# Records in jobs"
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 2747 4997 49630 44550 77770 88480
## [1] "Secs.Bib"
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.0800 0.2200 0.2400 0.2871 0.3300 0.7200
## [1] "Duration in seconds"
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 477 1350 9902 12770 18800 37540
## [1] "# of jobs: 430"
## [1] "# Records in jobs"
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.00 2.00 6.00 66.82 25.00 2243.00
## [1] "Secs.Bib"
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.140 1.922 3.130 6.241 8.748 55.000
## [1] "Duration in seconds"
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.00 9.25 32.00 114.20 114.80 2500.00
## Source: local data frame [20 x 8]
##
## Start BulkImpRule WeekDay Records Durationsecs Secs.Bib
## 1 11/5/2014 8:05 PANMUNFO Wed 1 55 55.00
## 2 11/5/2014 8:05 YANKEEFO Wed 3 137 45.67
## 3 11/19/2014 8:01 CASPLUS Wed 3 133 44.33
## 4 11/18/2014 8:05 BENFO Tue 4 160 40.00
## 5 11/18/2014 8:05 HARRASFO Tue 2 79 39.50
## 6 11/5/2014 8:05 ERASMUS Wed 4 155 38.75
## 7 11/19/2014 8:05 AUXFO Wed 1 35 35.00
## 8 11/5/2014 8:05 KINOFO Wed 4 138 34.50
## 9 11/18/2014 8:05 AUXFO Tue 6 161 26.83
## 10 11/14/2014 8:05 DKAFO Fri 3 73 24.33
## 11 11/14/2014 8:05 AMBASFO Fri 4 97 24.25
## 12 11/19/2014 8:00 YANKEEFO Wed 18 433 24.06
## 13 11/18/2014 8:05 AMBASFO Tue 7 165 23.57
## 14 11/19/2014 8:05 YANKEEFO Wed 6 140 23.33
## 15 11/19/2014 8:02 CASPLUS Wed 1 23 23.00
## 16 11/19/2014 8:05 PANMUNFO Wed 6 134 22.33
## 17 11/14/2014 8:05 COUTUKFO Fri 6 133 22.17
## 18 11/26/2014 8:05 COUTUKFO Wed 6 133 22.17
## 19 11/5/2014 8:05 DKAFO Wed 8 173 21.62
## 20 11/18/2014 8:05 YANKEEFO Tue 8 170 21.25
## Variables not shown: KeywordIndexing (chr), Added (int)
## Source: local data frame [20 x 7]
##
## KeywordIndexing BulkImpRule WeekDay Records Durationsecs Secs.Bib Added
## 1 no SERSOL Wed 12879 1086 0.08 0
## 2 yes HARRFULL Wed 7 1 0.14 0
## 3 no AUTHMERG Tue 11317 1754 0.15 5672
## 4 no BIBMAINT Fri 2930 477 0.16 0
## 5 no BIBMAINT Fri 63816 9902 0.16 0
## 6 no AUTHMERG Mon 11168 1955 0.18 6540
## 7 no BIBMAINT Tue 86363 16551 0.19 0
## 8 no BIBUPD Fri 46309 8802 0.19 0
## 9 no BIBMAINT Fri 4154 791 0.19 0
## 10 no BIBMAINT Thu 82602 15546 0.19 0
## 11 no BIBMAINT Mon 88484 18755 0.21 0
## 12 no BIBMAINT Sun 82715 18349 0.22 0
## 13 no AUTHMERG Mon 9153 2041 0.22 5502
## 14 no BIBMAINT Fri 4480 972 0.22 0
## 15 no BIBMAINT Sun 77771 17717 0.23 0
## 16 no BIBMAINT Thu 76578 17766 0.23 0
## 17 no BIBMAINT Fri 16291 3754 0.23 0
## 18 no AUTHMERG Tue 11393 2786 0.24 6368
## 19 no BIBMAINT Thu 77177 18213 0.24 0
## 20 no BIBMAINT Tue 4416 1077 0.24 0
## Source: local data frame [20 x 8]
##
## Start BulkImpRule WeekDay Records Secs.Bib Durationsecs
## 1 11/19/2014 7:53 HARRFULL Wed 7 0.14 1
## 2 11/5/2014 10:21 CAMBEIRO Wed 11 0.36 4
## 3 11/12/2014 10:26 BIBMAINT Wed 122 0.38 46
## 4 11/3/2014 7:29 AUTHMERG Mon 166 0.46 77
## 5 11/27/2014 23:03 BIBMAINT Thu 2243 0.50 1126
## 6 11/3/2014 22:50 BIBMAINT Mon 504 0.56 281
## 7 11/17/2014 11:01 SERSOLD Mon 596 0.59 350
## 8 11/3/2014 13:39 BIBSUP Mon 1452 0.63 912
## 9 11/25/2014 7:14 AUTHMERG Tue 399 0.65 260
## 10 11/11/2014 9:08 AUTHMERG Tue 74 0.70 52
## 11 11/17/2014 11:44 SERSOLD Mon 2021 0.71 1427
## 12 11/27/2014 16:22 BIBMAINT Thu 273 0.86 235
## 13 11/27/2014 19:22 BIBMAINT Thu 66 0.86 57
## 14 11/21/2014 15:48 BIBMAINT Fri 344 0.93 321
## 15 11/24/2014 19:30 BIBMAINT Mon 100 0.99 99
## 16 11/6/2014 8:05 YANKEEFO Thu 2 1.00 2
## 17 11/12/2014 16:13 CASPLUS Wed 1 1.00 1
## 18 11/17/2014 10:20 COUPDBFO Mon 1 1.00 1
## 19 11/17/2014 10:20 BIBSUP Mon 1 1.00 1
## 20 11/18/2014 8:05 CASALINI Tue 1 1.00 1
## Variables not shown: Added (int), KeywordIndexing (chr)