Testing Your Program

“The productive modeler simply assumes that software mistakes are inevitable and continually searches for them.”
- Railsback & Grimm

• Understand the difference between validation and verification.
• Seven common kinds of software errors that will blow your mind!
• Ten important techniques for finding and fixing software errors everyone should know.
• Understand why and how to document software tests.

• Typographical Errors
• Syntax Errors
• Misunderstanding Primitives
• Wrong Display Settings: Use resize-world in setup!
• Logic Errors: Runs but results incorrect
• Run-time Errors: No syntax or logic errors, but breaks on Go (sometimes)
• Formulation Errors: Incorrect assumptions & model decisions

Syntax Checking: Chunk-it and use skeleton code!

ifelse (xcor >= min-marriage-age)
[ show "If" ]
[ show "Else" ]

ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ show "If" ]
[ show "Else" ]

ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ set married? true ]
[ show "Else" ]

Visual Testing: Use visual cues of variables!

• Use scale-color to color turtles or patches based on their variables.
• Use label and plabel to check turtle or patch information.
• Use Agent and Patch Monitor.
• Use a smaller World to test things (actually, testing on smaller problems is a more general technique)
• Slow down the simulation and/or use a step button.

Print Statements

• For procedures:

to dostuff
  show "Starting procedure X"
  ; Do stuff
  show "Ending procedure X"
end

• For variables:

observer> show word "num turtles = " count turtles
observer: "num turtles = 0"

Code Reviews

• Reviewer's job:
  • Verification: Does the code match the ODD model formulation?
  • Fresh set of eyes can more easily find bugs (sometimes).
  • Make sure code is well-organized and easy to understand.
  • Write code as if someone will eventually read AND USE IT, even if you do not plan on it being used.

Statistical Analysis of File Output

Question: Is the probability butterflies move to the highest neighbor patch really \( q \)?

Answer: No. It is the approximate proportion \[ q + \frac{1-q}{8}. \]

For \( q = 0.4 \), we would expect the butterfly to move to the highest neighbor patch with probability 0.475.

Statistical Analysis of File Output

file-type
file-print
file-open
file-close

Statistical Analysis of File Output

mydata <- read.csv("TestOutput.csv", header=FALSE)
str(mydata)

'data.frame':   10000 obs. of  9 variables:
 $ V1: num  15.6 15.4 14.8 13.9 15 ...
 $ V2: num  14.8 14.8 15.4 14.4 14 ...
 $ V3: num  15.8 16.3 14.9 15.2 13 ...
 $ V4: num  15.6 14.9 14.6 13.9 13.9 ...
 $ V5: num  15.5 15.6 14.9 14.7 14.6 ...
 $ V6: num  15.5 14.7 16.1 15.4 13.8 ...
 $ V7: num  14.7 16.1 15.5 16 14.9 ...
 $ V8: num  16.3 16.9 13.9 13.8 14.9 ...
 $ V9: num  15.5 14.7 14.6 13.9 15 ...

Statistical Analysis of File Output

moved.to.highest <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,9]})

moved.to.highest <- as.integer(moved.to.highest)

Statistical Analysis of File Output

prop.test(sum(moved.to.highest), 10000, p = 0.475)

    1-sample proportions test with continuity correction

data:  sum(moved.to.highest) out of 10000, null probability 0.475
X-squared = 100.45, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.475
95 percent confidence interval:
 0.4151922 0.4346658
sample estimates:
     p 
0.4249 

Statistical Analysis of File Output

Discuss: Wait, what?!? Result doesn’t even contain 0.475 in the confidence interval (i.e. 0.475 isn’t even a plausible hypothesized proportion). Explain why the estimate is smaller than 0.475.

Hint: What if you are at the top of a hill?

Regardless, something is wrong. We are having a problem with verification, i.e. the program is not doing what it is supposed to do.

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)

Independent Reimplementation of Submodels

Test movement submodel.

'data.frame':   10000 obs. of  11 variables:
 $ V1 : num  15.6 16.9 17 18.9 18.2 ...
 $ V2 : num  15.6 16.4 19.1 19.2 16.9 ...
 $ V3 : num  15.5 14.9 16.3 18.3 19.7 ...
 $ V4 : num  14.7 16.1 16.9 19.7 17.7 ...
 $ V5 : num  14.8 15.5 18.4 18.4 18.9 ...
 $ V6 : num  16.3 15.6 17.8 19.8 19.1 ...
 $ V7 : num  15.8 17.7 17.5 20.5 18.4 ...
 $ V8 : num  15.5 17 18.3 17.7 17.5 ...
 $ V9 : num  0.0782 0.5299 0.1233 0.5341 0.7965 ...
 $ V10: num  0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 ...
 $ V11: num  16.3 17.7 19.1 18.3 18.2 ...

Independent Reimplementation of Submodels

moved.up <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,11]})

should.moved.up <- (mydata[,9] < mydata[,10])

(diff <- which((moved.up == FALSE) & (should.moved.up == TRUE)))

   [1]  172  173  176  192  205  223  224  225  263  278  296  304  318  319
  [15]  320  383  416  419  425  426  429  430  439  443  444  455  459  480
  [29]  498  514  527  528  565  566  578  597  606  616  617  618  619  627
  [43]  639  642  647  671  672  673  677  687  688  704  705  711  714  718
  [57]  722  723  726  737  738  739  740  762  763  821  826  827  832  833
  [71]  843  855  863  864  865  875  884  901  902  903  907  918  919  920
  [85]  921  922  923  933  939  940  941  968  969 1040 1052 1053 1056 1059
  [99] 1067 1070 1078 1114 1156 1157 1160 1165 1166 1169 1176 1177 1184 1189
 [113] 1214 1253 1289 1290 1300 1301 1308 1309 1319 1323 1324 1325 1335 1336
 [127] 1337 1338 1339 1340 1341 1342 1343 1344 1356 1360 1375 1378 1379 1380
 [141] 1397 1398 1406 1407 1412 1419 1420 1454 1468 1480 1484 1492 1499 1500
 [155] 1501 1504 1505 1506 1507 1510 1513 1528 1529 1530 1535 1536 1537 1540
 [169] 1541 1600 1613 1640 1643 1647 1650 1750 1760 1782 1804 1819 1822 1827
 [183] 1828 1835 1836 1837 1846 1866 1867 1868 1869 1874 1875 1876 1879 1880
 [197] 1881 1882 1885 1890 1907 1908 1909 1912 1913 1927 1937 1966 1967 1968
 [211] 1969 2011 2020 2021 2039 2040 2041 2042 2062 2065 2073 2074 2081 2082
 [225] 2085 2092 2093 2096 2097 2126 2127 2136 2144 2145 2146 2147 2156 2159
 [239] 2162 2163 2164 2165 2166 2177 2178 2228 2229 2232 2233 2246 2247 2248
 [253] 2251 2264 2275 2285 2288 2289 2290 2295 2296 2301 2306 2307 2345 2348
 [267] 2349 2350 2381 2382 2394 2397 2400 2401 2404 2413 2419 2422 2423 2427
 [281] 2439 2448 2449 2450 2459 2462 2463 2464 2465 2480 2500 2501 2505 2506
 [295] 2509 2510 2511 2514 2515 2531 2549 2554 2558 2559 2565 2568 2569 2574
 [309] 2577 2578 2583 2605 2606 2616 2632 2633 2634 2646 2647 2648 2649 2650
 [323] 2661 2662 2666 2667 2668 2669 2683 2702 2712 2726 2727 2728 2729 2730
 [337] 2731 2734 2743 2744 2747 2748 2756 2757 2758 2772 2773 2784 2789 2798
 [351] 2806 2807 2840 2841 2842 2873 2878 2883 2894 2913 2930 2933 2934 2935
 [365] 2936 2951 2954 2972 2983 3000 3013 3029 3038 3075 3076 3091 3094 3095
 [379] 3099 3100 3105 3106 3116 3128 3150 3153 3156 3157 3158 3159 3174 3183
 [393] 3184 3189 3196 3202 3203 3204 3214 3215 3218 3223 3249 3282 3298 3299
 [407] 3300 3301 3302 3326 3352 3367 3368 3369 3386 3387 3395 3412 3448 3449
 [421] 3450 3457 3458 3459 3478 3488 3491 3498 3507 3508 3509 3510 3554 3561
 [435] 3564 3576 3577 3578 3586 3591 3592 3593 3609 3610 3616 3619 3628 3629
 [449] 3630 3631 3636 3649 3652 3669 3670 3671 3672 3684 3756 3759 3760 3761
 [463] 3762 3765 3768 3789 3792 3793 3804 3807 3808 3809 3823 3850 3856 3857
 [477] 3864 3881 3882 3883 3886 3887 3912 3936 3937 3940 3941 3944 3951 3956
 [491] 3957 3993 3997 3998 4024 4035 4043 4044 4047 4054 4097 4098 4106 4107
 [505] 4164 4165 4166 4169 4172 4173 4181 4196 4197 4198 4210 4227 4269 4281
 [519] 4294 4297 4298 4299 4302 4305 4306 4307 4308 4309 4310 4313 4332 4333
 [533] 4336 4337 4347 4348 4351 4371 4377 4378 4396 4397 4401 4402 4412 4413
 [547] 4414 4415 4416 4419 4420 4429 4430 4433 4434 4435 4436 4437 4441 4444
 [561] 4445 4451 4452 4455 4467 4481 4489 4490 4491 4503 4504 4507 4508 4511
 [575] 4512 4531 4532 4535 4536 4548 4553 4564 4565 4582 4640 4645 4678 4679
 [589] 4682 4683 4684 4700 4701 4728 4729 4732 4740 4741 4744 4745 4758 4759
 [603] 4772 4773 4774 4790 4791 4797 4847 4860 4861 4868 4869 4870 4871 4872
 [617] 4873 4874 4877 4896 4897 4921 4926 4931 4932 4939 4965 4977 4982 4988
 [631] 4989 4990 4991 4992 4993 5005 5006 5007 5010 5032 5046 5062 5076 5085
 [645] 5089 5107 5139 5147 5171 5213 5229 5230 5235 5238 5239 5292 5297 5298
 [659] 5299 5300 5330 5331 5343 5362 5366 5367 5368 5369 5375 5376 5379 5384
 [673] 5395 5396 5397 5405 5406 5413 5418 5419 5420 5427 5430 5431 5463 5464
 [687] 5474 5479 5484 5497 5508 5512 5513 5519 5520 5521 5522 5543 5546 5549
 [701] 5550 5551 5556 5557 5558 5562 5568 5575 5576 5603 5619 5632 5640 5677
 [715] 5701 5712 5719 5747 5768 5774 5775 5778 5785 5791 5808 5809 5865 5870
 [729] 5871 5872 5898 5911 5917 5922 5923 5926 5929 5954 5955 5958 5959 5964
 [743] 5967 5972 5973 5974 5993 6000 6005 6006 6007 6008 6009 6010 6011 6012
 [757] 6013 6020 6047 6048 6059 6064 6074 6107 6120 6123 6124 6131 6153 6162
 [771] 6163 6164 6167 6170 6171 6198 6199 6200 6201 6202 6208 6263 6303 6326
 [785] 6343 6379 6394 6395 6406 6412 6461 6477 6483 6484 6491 6518 6519 6520
 [799] 6541 6584 6585 6610 6611 6612 6621 6622 6630 6648 6672 6688 6712 6717
 [813] 6738 6739 6740 6743 6748 6751 6752 6753 6768 6778 6779 6787 6801 6828
 [827] 6829 6837 6848 6861 6862 6880 6892 6893 6923 6928 6931 6932 6936 6937
 [841] 6938 6961 7005 7040 7045 7055 7056 7057 7060 7070 7083 7084 7085 7086
 [855] 7095 7106 7121 7137 7152 7177 7221 7234 7235 7240 7245 7250 7305 7308
 [869] 7318 7324 7325 7326 7351 7352 7353 7367 7368 7378 7381 7401 7418 7431
 [883] 7434 7435 7436 7437 7438 7442 7443 7444 7447 7450 7451 7458 7463 7491
 [897] 7492 7526 7541 7542 7554 7555 7563 7568 7593 7594 7606 7613 7614 7626
 [911] 7637 7638 7642 7643 7644 7649 7650 7655 7656 7657 7678 7679 7682 7683
 [925] 7693 7722 7730 7731 7732 7733 7739 7740 7751 7752 7764 7778 7779 7780
 [939] 7781 7812 7813 7824 7825 7836 7845 7846 7868 7869 7870 7871 7887 7899
 [953] 7904 7905 7923 7928 7931 7949 7952 7961 7967 7968 7969 7970 7992 8000
 [967] 8001 8004 8005 8011 8030 8031 8034 8035 8040 8043 8044 8045 8046 8057
 [981] 8060 8086 8094 8101 8106 8107 8130 8141 8146 8149 8150 8157 8169 8170
 [995] 8173 8174 8196 8197 8198 8199 8206 8209 8210 8219 8222 8225 8254 8262
[1009] 8276 8281 8282 8292 8293 8294 8318 8319 8320 8328 8337 8338 8391 8403
[1023] 8472 8473 8485 8488 8489 8490 8497 8498 8499 8502 8512 8513 8514 8515
[1037] 8516 8523 8541 8542 8546 8547 8550 8555 8575 8576 8585 8588 8604 8610
[1051] 8618 8627 8664 8667 8668 8677 8716 8731 8732 8739 8742 8743 8769 8774
[1065] 8802 8808 8809 8812 8818 8819 8820 8827 8853 8910 8917 8918 8919 8920
[1079] 8934 8941 8942 8943 8944 8945 8946 8947 8951 8952 8955 8963 8964 8978
[1093] 8979 8984 8985 8986 8992 8993 9015 9016 9031 9046 9049 9057 9069 9079
[1107] 9080 9085 9089 9110 9115 9124 9129 9130 9131 9145 9146 9157 9168 9181
[1121] 9190 9195 9200 9226 9227 9232 9235 9251 9261 9262 9267 9276 9277 9278
[1135] 9289 9290 9291 9304 9313 9318 9319 9320 9324 9343 9361 9365 9368 9369
[1149] 9383 9404 9405 9406 9407 9423 9424 9425 9428 9438 9439 9452 9453 9466
[1163] 9488 9498 9499 9505 9527 9528 9541 9542 9549 9557 9558 9592 9601 9602
[1177] 9605 9608 9614 9615 9626 9634 9635 9651 9663 9668 9669 9670 9676 9704
[1191] 9752 9777 9792 9793 9816 9819 9820 9821 9822 9828 9832 9833 9847 9848
[1205] 9849 9850 9851 9857 9860 9914 9915 9918 9921 9928 9929 9930 9931 9937
[1219] 9938 9967 9977 9988 9989

Independent Reimplementation of Submodels

    V1       V2       V3 V4
172 99 98.58579 98.58579 99

    V5 V6       V7       V8 V11
172 99 99 98.58579 98.58579 100

Discuss: Given that V1-V8 are the neighbors elevation, and V11 is the elevation of the patch moved to, what does this tell you?

Answer: If at top of the hill, turtle stays put!

Independent Reimplementation of Submodels

Discuss: Does this explain \( q\neq 0.475 \) discrepancy from earlier?

Answer: Yes! If at top of hill, will not be equal to max of neighbors.

Independent Reimplementation of Submodels

From ODD Submodels:

“..to 'move uphill' is defined specifically as moving to the neighbor patch that has the highest elevation; if two patches have the same elevation, one is chosen randomly. 'Move randomly' is defined as moving to one of the neighboring patches, with equal probability of choosing any patch. 'Neighbor patches' are the eight patches surrounding the butterfly's current patch.”

OOPS!

Independent Reimplementation of Submodels

How to fix? Change

[uphill elevation]

to

[move-to max-one-of neighbors [elevation]]

This was a “Misunderstanding Primitives” error!!!