Chapter 6: Testing Your Program

April 11, 2024

Chapter 6: Learning Objectives

“The productive modeler simply assumes that software mistakes are inevitable and continually searches for them.”

- Railsback & Grimm

  • Understand the difference between validation and verification.
  • Seven common kinds of software errors that will blow your mind!
  • Ten important techniques for finding and fixing software errors everyone should know.
  • Understand why and how to document software tests.

Chapter 6: Validation vs Verification

Part I. Common Kinds of Errors

  • Typographical Errors
  • Syntax Errors
  • Misunderstanding Primitives
  • Wrong Display Settings: Use resize-world in setup!
  • Logic Errors: Runs but results incorrect
  • Run-time Errors: No syntax or logic errors, but breaks on Go (sometimes)
  • Formulation Errors: Incorrect assumptions & model decisions

Part II. Debugging Techniques

Syntax Checking: Chunk-it and use skeleton code!

ifelse (xcor >= min-marriage-age)
[ show "If" ]
[ show "Else" ]
ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ show "If" ]
[ show "Else" ]
ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ set married? true ]
[ show "Else" ]

Part II. Debugging Techniques

Visual Testing: Use visual cues of variables!

  • Use scale-color to color turtles or patches based on their variables.
  • Use label and plabel to check turtle or patch information.
  • Use Agent and Patch Monitor.
  • Use a smaller World to test things (actually, testing on smaller problems is a more general technique)
  • Slow down the simulation and/or use a step button.

Part II. Debugging Techniques

Print Statements

  • For procedures:
to dostuff
  show "Starting procedure X"
  ; Do stuff
  show "Ending procedure X"
end
  • For variables:
observer> show word "num turtles = " count turtles
observer: "num turtles = 0"

Part II. Debugging Techniques

Spot Tests with Agent Monitors

Part II. Debugging Techniques

Stress Tests

Use parameters and initial data at the extreme values, and possibly outside normal ranges (e.g. \(q = 1.0\))

Part II. Debugging Techniques

Test Procedures

and

Test Programs

Test Program

Part II. Debugging Techniques

Code Reviews

  • Reviewer’s job:
    • Verification: Does the code match the ODD model formulation?
    • Fresh set of eyes can more easily find bugs (sometimes).
    • Make sure code is well-organized and easy to understand.
    • Write code as if someone will eventually read AND USE IT, even if you do not plan on it being used.

Part II. Debugging Techniques

Statistical Analysis of File Output

Question: Is the probability butterflies move to the highest neighbor patch really \(q\)?

Answer: No. It is the approximate proportion \[ q + \frac{1-q}{8}. \]

For \(q = 0.4\), we would expect the butterfly to move to the highest neighbor patch with probability 0.475.

Part II. Debugging Techniques

Statistical Analysis of File Output

file-type
file-print
file-open
file-close

Part II. Debugging Techniques

Statistical Analysis of File Output

mydata <- read.csv("TestOutput.csv", header=FALSE)
str(mydata)
'data.frame':   10000 obs. of  9 variables:
 $ V1: num  15.8 17.7 16.1 19.1 20.4 ...
 $ V2: num  15.6 17 17 16.9 19.6 ...
 $ V3: num  14.7 15.5 17.5 18.4 19.1 ...
 $ V4: num  15.5 15.6 17.7 18.2 18.3 ...
 $ V5: num  15.6 14.9 18.3 18.9 18.9 ...
 $ V6: num  14.8 16.4 16.3 17.7 20.5 ...
 $ V7: num  15.5 16.1 16.8 19.7 21.1 ...
 $ V8: num  16.3 16.9 15.5 17.5 19.8 ...
 $ V9: num  16.3 16.9 18.3 19.7 19.6 ...

Part II. Debugging Techniques

Statistical Analysis of File Output

moved.to.highest <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,9]})

moved.to.highest <- as.integer(moved.to.highest)

Part II. Debugging Techniques

Statistical Analysis of File Output

prop.test(sum(moved.to.highest), 10000, p = 0.475)

    1-sample proportions test with continuity correction

data:  sum(moved.to.highest) out of 10000, null probability 0.475
X-squared = 113.28, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.475
95 percent confidence interval:
 0.4121028 0.4315576
sample estimates:
     p 
0.4218

Part II. Debugging Techniques

Statistical Analysis of File Output

Discuss: Wait, what?!? Result doesn’t even contain 0.475 in the confidence interval (i.e. 0.475 isn’t even a plausible hypothesized proportion). Explain why the estimate is smaller than 0.475.

Hint: What if you are at the top of a hill?

Regardless, something is wrong. We are having a problem with verification, i.e. the program is not doing what it is supposed to do.

Part II. Debugging Techniques

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)

Part II. Debugging Techniques

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)
'data.frame':   10000 obs. of  11 variables:
 $ V1 : num  15.6 15.6 16.1 17.5 16.9 ...
 $ V2 : num  15.5 16.1 13.9 16.3 18.9 ...
 $ V3 : num  14.8 14.9 15.4 16.8 17.7 ...
 $ V4 : num  14.7 15.4 14.6 15.5 16.8 ...
 $ V5 : num  15.8 14.7 14.9 14.7 18.3 ...
 $ V6 : num  16.3 16.9 14.8 16 18.2 ...
 $ V7 : num  15.6 16.3 15.5 16.9 16.1 ...
 $ V8 : num  15.5 14.8 14.9 15.4 17.4 ...
 $ V9 : num  0.6357 0.8528 0.0527 0.3981 0.4959 ...
 $ V10: num  0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 ...
 $ V11: num  15.5 14.7 16.1 17.5 18.9 ...

Part II. Debugging Techniques

Independent Reimplementation of Submodels

moved.up <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,11]})

should.moved.up <- (mydata[,9] < mydata[,10])

(diff <- which((moved.up == FALSE) & (should.moved.up == TRUE)))
   [1]  180  181  182  185  186  213  258  264  267  272  273  274  275  276
  [15]  284  321  322  327  328  343  344  347  350  351  352  355  358  361
  [29]  366  372  375  414  417  418  421  426  431  434  437  440  441  442
  [43]  447  458  495  498  499  500  501  502  515  516  517  535  536  537
  [57]  538  543  579  595  604  616  623  640  646  659  663  664  682  683
  [71]  684  695  696  704  705  706  707  708  709  718  719  724  740  745
  [85]  746  751  763  788  792  802  803  818  819  832  849  850  851  869
  [99]  870  882  883  890  897  900  908  931  932  933  975  984  990  991
 [113] 1004 1007 1028 1029 1030 1072 1073 1076 1082 1083 1086 1087 1088 1089
 [127] 1095 1098 1101 1102 1105 1111 1116 1117 1122 1127 1128 1139 1145 1150
 [141] 1151 1156 1157 1158 1159 1217 1225 1226 1227 1228 1233 1234 1244 1250
 [155] 1262 1263 1264 1265 1266 1273 1276 1277 1278 1281 1361 1362 1363 1368
 [169] 1369 1372 1377 1386 1424 1456 1457 1460 1525 1529 1530 1547 1552 1559
 [183] 1560 1571 1572 1587 1604 1643 1649 1654 1655 1656 1657 1674 1677 1678
 [197] 1679 1680 1694 1695 1696 1699 1702 1709 1710 1718 1723 1724 1791 1794
 [211] 1795 1802 1814 1815 1816 1821 1844 1847 1898 1899 1900 1903 1908 1913
 [225] 1931 1932 1933 1936 1952 1962 1966 1967 1994 1997 2034 2056 2059 2060
 [239] 2061 2062 2063 2064 2080 2081 2103 2117 2120 2121 2122 2127 2130 2131
 [253] 2134 2137 2138 2139 2143 2144 2145 2200 2201 2205 2208 2262 2263 2281
 [267] 2287 2301 2326 2327 2328 2331 2332 2333 2346 2347 2367 2380 2385 2396
 [281] 2397 2402 2434 2445 2446 2481 2484 2485 2488 2493 2496 2510 2511 2512
 [295] 2513 2516 2526 2553 2582 2590 2595 2599 2618 2626 2627 2628 2629 2653
 [309] 2684 2689 2694 2699 2726 2727 2728 2737 2738 2741 2765 2775 2815 2831
 [323] 2871 2905 2906 2913 2916 2924 2925 2965 2968 3000 3001 3002 3008 3023
 [337] 3024 3055 3056 3069 3112 3113 3116 3117 3125 3132 3133 3139 3164 3169
 [351] 3172 3173 3196 3197 3198 3227 3228 3229 3237 3240 3269 3272 3355 3358
 [365] 3368 3369 3377 3380 3384 3387 3394 3400 3407 3412 3413 3414 3452 3457
 [379] 3467 3468 3469 3470 3473 3476 3477 3488 3494 3495 3496 3499 3514 3515
 [393] 3519 3522 3523 3527 3537 3546 3547 3562 3565 3566 3567 3568 3569 3570
 [407] 3580 3653 3654 3655 3656 3657 3679 3680 3683 3684 3694 3735 3736 3737
 [421] 3742 3743 3754 3762 3763 3764 3767 3807 3808 3809 3830 3831 3832 3833
 [435] 3834 3841 3854 3859 3860 3861 3862 3863 3868 3869 3870 3878 3879 3880
 [449] 3883 3910 3913 3914 3917 3918 3929 3941 3952 3953 3958 4003 4010 4028
 [463] 4029 4038 4060 4061 4068 4081 4082 4083 4086 4089 4098 4099 4110 4111
 [477] 4117 4130 4131 4132 4133 4134 4148 4167 4205 4208 4223 4243 4246 4275
 [491] 4276 4277 4281 4282 4283 4290 4338 4341 4344 4345 4354 4355 4356 4357
 [505] 4358 4361 4364 4375 4378 4379 4380 4411 4412 4413 4422 4423 4424 4429
 [519] 4438 4445 4446 4450 4489 4492 4493 4498 4505 4518 4519 4524 4525 4541
 [533] 4542 4543 4549 4560 4582 4621 4622 4623 4650 4668 4669 4682 4683 4736
 [547] 4739 4757 4760 4761 4762 4768 4785 4794 4797 4810 4833 4840 4864 4867
 [561] 4897 4913 4922 4923 4934 4935 4963 4970 4973 4974 4985 4990 4993 5009
 [575] 5014 5015 5016 5017 5022 5036 5039 5040 5041 5042 5043 5047 5050 5055
 [589] 5060 5065 5068 5071 5085 5086 5094 5095 5096 5099 5103 5126 5132 5192
 [603] 5193 5194 5195 5205 5249 5250 5253 5254 5255 5263 5280 5281 5282 5290
 [617] 5296 5299 5306 5311 5345 5346 5347 5351 5352 5356 5370 5373 5418 5419
 [631] 5420 5452 5482 5487 5498 5501 5506 5507 5535 5538 5541 5557 5567 5568
 [645] 5569 5572 5582 5598 5609 5610 5611 5612 5613 5636 5637 5650 5653 5669
 [659] 5672 5676 5681 5687 5704 5705 5730 5731 5732 5733 5734 5743 5760 5761
 [673] 5772 5773 5780 5812 5821 5822 5823 5824 5825 5826 5827 5828 5829 5833
 [687] 5848 5870 5877 5882 5908 5919 5934 5935 5936 5941 5948 5955 5956 5957
 [701] 5960 5963 5964 5965 5968 5969 5970 5971 5976 6026 6033 6034 6035 6036
 [715] 6063 6064 6065 6076 6080 6081 6082 6107 6136 6140 6141 6154 6155 6173
 [729] 6176 6177 6182 6203 6206 6216 6237 6238 6293 6299 6300 6301 6304 6327
 [743] 6328 6329 6334 6358 6385 6421 6424 6425 6435 6436 6439 6457 6458 6461
 [757] 6467 6470 6492 6511 6516 6517 6520 6530 6531 6535 6536 6537 6577 6578
 [771] 6579 6580 6583 6588 6589 6590 6602 6645 6646 6651 6682 6693 6715 6716
 [785] 6748 6749 6760 6761 6774 6775 6793 6794 6795 6796 6806 6807 6810 6833
 [799] 6845 6848 6849 6850 6851 6856 6868 6880 6881 6889 6890 6891 6894 6898
 [813] 6899 6903 6908 6923 6948 6949 6950 6951 6968 6969 6979 6980 6981 6990
 [827] 6991 6994 7012 7013 7021 7022 7025 7033 7034 7035 7040 7044 7071 7072
 [841] 7073 7089 7103 7104 7107 7115 7136 7137 7138 7146 7149 7150 7151 7162
 [855] 7163 7233 7234 7243 7244 7245 7246 7257 7260 7261 7262 7276 7281 7282
 [869] 7283 7288 7289 7323 7338 7341 7344 7347 7389 7390 7419 7437 7438 7446
 [883] 7447 7469 7470 7516 7537 7538 7541 7578 7582 7589 7598 7603 7613 7614
 [897] 7615 7616 7619 7623 7626 7627 7630 7659 7687 7688 7691 7692 7706 7707
 [911] 7723 7756 7757 7760 7761 7783 7784 7802 7879 7880 7884 7895 7896 7899
 [925] 7900 7901 7926 7954 7982 7992 8014 8017 8018 8022 8035 8036 8037 8038
 [939] 8039 8040 8047 8096 8099 8117 8130 8131 8132 8133 8170 8177 8187 8191
 [953] 8192 8193 8194 8195 8196 8197 8198 8201 8213 8218 8231 8234 8241 8242
 [967] 8252 8286 8289 8328 8329 8345 8346 8347 8348 8351 8356 8370 8371 8375
 [981] 8376 8379 8382 8385 8386 8389 8418 8419 8433 8434 8435 8447 8448 8465
 [995] 8466 8467 8468 8469 8477 8478 8479 8480 8481 8490 8491 8494 8497 8498
[1009] 8499 8506 8522 8525 8535 8544 8545 8558 8559 8571 8572 8587 8588 8589
[1023] 8590 8597 8600 8601 8606 8607 8636 8647 8683 8684 8685 8702 8703 8704
[1037] 8707 8710 8713 8720 8735 8738 8741 8742 8745 8746 8766 8767 8776 8777
[1051] 8778 8785 8794 8821 8829 8832 8836 8844 8849 8853 8859 8866 8872 8882
[1065] 8892 8893 8903 8904 8907 8908 8909 8910 8911 8918 8942 9003 9006 9023
[1079] 9027 9066 9067 9068 9078 9079 9085 9090 9101 9106 9107 9128 9129 9134
[1093] 9137 9142 9143 9190 9191 9216 9217 9218 9222 9231 9232 9239 9240 9247
[1107] 9255 9258 9259 9283 9286 9293 9296 9297 9317 9318 9319 9323 9324 9325
[1121] 9388 9389 9403 9404 9419 9420 9421 9422 9423 9438 9450 9451 9452 9453
[1135] 9461 9473 9486 9490 9491 9494 9504 9550 9559 9593 9594 9595 9617 9636
[1149] 9663 9668 9669 9672 9673 9674 9679 9680 9705 9706 9707 9721 9722 9726
[1163] 9729 9734 9741 9764 9767 9778 9781 9805 9821 9848 9854 9887 9930 9945
[1177] 9946 9947 9960 9961 9964 9967 9968 9973 9980

Part II. Debugging Techniques

Independent Reimplementation of Submodels

mydata[diff[1],1:4]
    V1       V2       V3       V4
180 99 98.58579 98.58579 98.58579
mydata[diff[1],c(5:8, 11)]
          V5 V6 V7 V8 V11
180 98.58579 99 99 99 100

Discuss: Given that V1-V8 are the neighbors elevation, and V11 is the elevation of the patch moved to, what does this tell you?

Answer: If at top of the hill, turtle stays put!

Part II. Debugging Techniques

Independent Reimplementation of Submodels

Discuss: Does this explain \(q\neq 0.475\) discrepancy from earlier?

Answer: Yes! If at top of hill, will not be equal to max of neighbors.

Part II. Debugging Techniques

Independent Reimplementation of Submodels

From ODD Submodels:

“..to ‘move uphill’ is defined specifically as moving to the neighbor patch that has the highest elevation; if two patches have the same elevation, one is chosen randomly. ‘Move randomly’ is defined as moving to one of the neighboring patches, with equal probability of choosing any patch. ‘Neighbor patches’ are the eight patches surrounding the butterfly’s current patch.”

OOPS!

Part II. Debugging Techniques

Independent Reimplementation of Submodels

doh

Part II. Debugging Techniques

Independent Reimplementation of Submodels

How to fix? Change

[uphill elevation]

to

[move-to max-one-of neighbors [elevation]]

This was a “Misunderstanding Primitives” error!!!

Intro to Quantitative Biology, Spring 2024