Chapter 6: Testing Your Program

April 10, 2023

Chapter 6: Learning Objectives

“The productive modeler simply assumes that software mistakes are inevitable and continually searches for them.” - Railsback & Grimm

  • Understand the difference between validation and verification.
  • Seven common kinds of software errors that will blow your mind!
  • Ten important techniques for finding and fixing software errors everyone should know.
  • Understand why and how to document software tests.

Chapter 6: Validation vs Verification

Part I. Common Kinds of Errors

  • Typographical Errors
  • Syntax Errors
  • Misunderstanding Primitives
  • Wrong Display Settings: Use resize-world in setup!
  • Logic Errors: Runs but results incorrect
  • Run-time Errors: No syntax or logic errors, but breaks on Go (sometimes)
  • Formulation Errors: Incorrect assumptions & model decisions

Part II. Debugging Techniques

Syntax Checking: Chunk-it and use skeleton code!

ifelse (xcor >= min-marriage-age)
[ show "If" ]
[ show "Else" ]
ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ show "If" ]
[ show "Else" ]
ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ set married? true ]
[ show "Else" ]

Part II. Debugging Techniques

Visual Testing: Use visual cues of variables!

  • Use scale-color to color turtles or patches based on their variables.
  • Use label and plabel to check turtle or patch information.
  • Use Agent and Patch Monitor.
  • Use a smaller World to test things (actually, testing on smaller problems is a more general technique)
  • Slow down the simulation and/or use a step button.

Part II. Debugging Techniques

Print Statements

  • For procedures:
to dostuff
  show "Starting procedure X"
  ; Do stuff
  show "Ending procedure X"
end
  • For variables:
observer> show word "num turtles = " count turtles
observer: "num turtles = 0"

Part II. Debugging Techniques

Spot Tests with Agent Monitors

Part II. Debugging Techniques

Stress Tests

Use parameters and initial data at the extreme values, and possibly outside normal ranges (e.g. \(q = 1.0\))

Part II. Debugging Techniques

Test Procedures

and

Test Programs

Test Program

Part II. Debugging Techniques

Code Reviews

  • Reviewer’s job:
    • Verification: Does the code match the ODD model formulation?
    • Fresh set of eyes can more easily find bugs (sometimes).
    • Make sure code is well-organized and easy to understand.
    • Write code as if someone will eventually read AND USE IT, even if you do not plan on it being used.

Part II. Debugging Techniques

Statistical Analysis of File Output

Question: Is the probability butterflies move to the highest neighbor patch really \(q\)?

Answer: No. It is the approximate proportion \[ q + \frac{1-q}{8}. \]

For \(q = 0.4\), we would expect the butterfly to move to the highest neighbor patch with probability 0.475.

Part II. Debugging Techniques

Statistical Analysis of File Output

file-type
file-print
file-open
file-close

Part II. Debugging Techniques

Statistical Analysis of File Output

mydata <- read.csv("TestOutput.csv", header=FALSE)
str(mydata)
'data.frame':   10000 obs. of  9 variables:
 $ V1: num  15.8 17.7 16.1 19.1 20.4 ...
 $ V2: num  15.6 17 17 16.9 19.6 ...
 $ V3: num  14.7 15.5 17.5 18.4 19.1 ...
 $ V4: num  15.5 15.6 17.7 18.2 18.3 ...
 $ V5: num  15.6 14.9 18.3 18.9 18.9 ...
 $ V6: num  14.8 16.4 16.3 17.7 20.5 ...
 $ V7: num  15.5 16.1 16.8 19.7 21.1 ...
 $ V8: num  16.3 16.9 15.5 17.5 19.8 ...
 $ V9: num  16.3 16.9 18.3 19.7 19.6 ...

Part II. Debugging Techniques

Statistical Analysis of File Output

moved.to.highest <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,9]})

moved.to.highest <- as.integer(moved.to.highest)

Part II. Debugging Techniques

Statistical Analysis of File Output

prop.test(sum(moved.to.highest), 10000, p = 0.475)

    1-sample proportions test with continuity correction

data:  sum(moved.to.highest) out of 10000, null probability 0.475
X-squared = 113.28, df = 1, p-value < 2.2e-16
alternative hypothesis: true p is not equal to 0.475
95 percent confidence interval:
 0.4121028 0.4315576
sample estimates:
     p 
0.4218

Part II. Debugging Techniques

Statistical Analysis of File Output

Discuss: Wait, what?!? Result doesn’t even contain 0.475 in the confidence interval (i.e. 0.475 isn’t even a plausible hypothesized proportion). Explain why the estimate is smaller than 0.475.

Hint: What if you are at the top of a hill?

Regardless, something is wrong. We are having a problem with verification, i.e. the program is not doing what it is supposed to do.

Part II. Debugging Techniques

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)

Part II. Debugging Techniques

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)
'data.frame':   10000 obs. of  11 variables:
 $ V1 : num  15.6 16.1 17 17.1 20.6 ...
 $ V2 : num  14.7 16.4 15.5 19.2 18.5 ...
 $ V3 : num  16.3 15.6 15.6 17 18.4 ...
 $ V4 : num  15.5 17 16.5 18.5 19.9 ...
 $ V5 : num  15.6 15.5 17.1 17.7 19.1 ...
 $ V6 : num  15.8 16.9 17.8 18.4 19.8 ...
 $ V7 : num  14.8 17.7 15.7 16.4 19.3 ...
 $ V8 : num  15.5 14.9 16.3 17.9 17.8 ...
 $ V9 : num  0.53 0.57 0.537 0.178 0.146 ...
 $ V10: num  0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 ...
 $ V11: num  16.3 16.4 17.8 19.2 20.6 ...

Part II. Debugging Techniques

Independent Reimplementation of Submodels

moved.up <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,11]})

should.moved.up <- (mydata[,9] < mydata[,10])

(diff <- which((moved.up == FALSE) & (should.moved.up == TRUE)))
   [1]  197  198  214  217  224  233  234  238  241  244  253  254  257  262
  [15]  263  264  265  272  273  274  277  278  284  285  293  294  295  296
  [29]  297  315  316  317  325  336  337  354  355  356  357  386  400  405
  [43]  406  422  423  426  463  467  474  475  481  517  518  519  524  525
  [57]  536  549  609  612  613  614  618  619  620  628  629  650  651  652
  [71]  653  681  688  692  705  706  725  753  754  776  777  783  788  789
  [85]  803  808  811  849  874  900  901  923  926  939  940  941  942  952
  [99]  953  956  957  961  964  965  986  987  995  996  997  998  999 1017
 [113] 1050 1087 1090 1091 1092 1113 1116 1131 1146 1147 1150 1153 1154 1157
 [127] 1187 1192 1208 1209 1210 1215 1295 1296 1297 1298 1299 1345 1354 1355
 [141] 1356 1357 1360 1361 1364 1401 1422 1430 1431 1432 1447 1454 1455 1460
 [155] 1468 1488 1507 1532 1535 1538 1547 1553 1563 1580 1605 1610 1621 1622
 [169] 1623 1632 1635 1639 1683 1684 1685 1696 1702 1703 1706 1711 1721 1724
 [183] 1744 1750 1751 1756 1769 1772 1801 1808 1809 1810 1811 1826 1846 1863
 [197] 1866 1873 1874 1875 1876 1890 1896 1907 1928 1931 1932 1933 1946 1949
 [211] 1963 1964 1965 1980 1981 1986 1999 2002 2005 2008 2013 2059 2062 2113
 [225] 2123 2139 2140 2141 2169 2192 2193 2201 2218 2228 2229 2230 2231 2232
 [239] 2235 2249 2272 2273 2287 2300 2301 2325 2351 2354 2355 2358 2359 2368
 [253] 2411 2412 2415 2428 2451 2461 2462 2473 2476 2477 2478 2479 2480 2483
 [267] 2488 2489 2493 2511 2534 2542 2543 2544 2547 2548 2551 2562 2565 2566
 [281] 2567 2568 2569 2570 2576 2577 2592 2593 2594 2599 2602 2612 2625 2639
 [295] 2644 2645 2646 2647 2648 2649 2655 2658 2732 2739 2740 2743 2752 2759
 [309] 2760 2769 2770 2799 2810 2844 2851 2866 2867 2877 2878 2893 2913 2927
 [323] 2928 2942 3012 3013 3014 3042 3047 3048 3061 3071 3102 3110 3115 3116
 [337] 3136 3137 3140 3141 3142 3163 3164 3168 3169 3170 3176 3177 3188 3196
 [351] 3226 3227 3252 3281 3282 3285 3288 3347 3375 3376 3377 3386 3406 3407
 [365] 3410 3438 3439 3442 3451 3452 3473 3474 3475 3478 3492 3496 3497 3517
 [379] 3524 3525 3528 3531 3532 3533 3534 3541 3552 3553 3564 3577 3580 3591
 [393] 3592 3593 3594 3612 3645 3650 3651 3659 3665 3666 3667 3668 3675 3684
 [407] 3696 3697 3698 3702 3703 3784 3822 3852 3862 3874 3879 3880 3887 3888
 [421] 3893 3901 3904 3907 3935 3936 3958 3984 3991 3996 3997 4000 4001 4004
 [435] 4007 4023 4033 4047 4048 4051 4052 4053 4054 4055 4058 4061 4062 4063
 [449] 4066 4091 4097 4098 4102 4116 4133 4140 4141 4148 4149 4153 4154 4162
 [463] 4176 4182 4183 4189 4190 4245 4246 4247 4257 4312 4313 4314 4320 4350
 [477] 4356 4367 4368 4369 4395 4400 4409 4412 4419 4420 4421 4450 4451 4452
 [491] 4453 4454 4462 4463 4469 4470 4478 4479 4480 4481 4482 4483 4484 4498
 [505] 4503 4506 4511 4517 4551 4552 4578 4579 4582 4621 4631 4644 4664 4665
 [519] 4666 4671 4672 4683 4690 4691 4694 4701 4720 4721 4722 4729 4742 4743
 [533] 4775 4776 4777 4782 4799 4803 4806 4811 4812 4819 4820 4860 4865 4866
 [547] 4867 4868 4869 4870 4873 4874 4905 4906 4909 4923 4928 4929 4935 4964
 [561] 4995 4996 4997 5002 5003 5017 5020 5030 5036 5037 5038 5050 5053 5054
 [575] 5055 5056 5077 5086 5087 5133 5134 5135 5140 5141 5142 5147 5148 5154
 [589] 5173 5183 5184 5185 5186 5189 5192 5202 5205 5206 5209 5210 5218 5222
 [603] 5225 5226 5253 5258 5259 5262 5263 5289 5290 5307 5310 5311 5314 5319
 [617] 5356 5361 5365 5366 5367 5372 5389 5406 5429 5432 5435 5436 5442 5456
 [631] 5476 5481 5482 5485 5505 5506 5509 5510 5518 5519 5548 5552 5567 5568
 [645] 5594 5599 5606 5630 5635 5669 5670 5675 5713 5714 5715 5719 5720 5725
 [659] 5730 5733 5734 5735 5762 5775 5778 5788 5797 5798 5799 5800 5801 5804
 [673] 5812 5813 5814 5815 5818 5840 5847 5850 5851 5852 5863 5864 5877 5878
 [687] 5881 5882 5895 5896 5906 5907 5918 5919 5922 5926 5973 5988 5989 6001
 [701] 6002 6007 6015 6016 6023 6024 6025 6039 6055 6058 6061 6062 6065 6083
 [715] 6112 6113 6114 6121 6124 6161 6167 6175 6176 6209 6233 6234 6235 6253
 [729] 6256 6296 6300 6301 6302 6303 6304 6307 6316 6317 6318 6319 6322 6323
 [743] 6352 6360 6361 6362 6365 6366 6367 6368 6381 6382 6385 6401 6402 6406
 [757] 6409 6410 6436 6443 6471 6485 6491 6492 6502 6512 6523 6557 6558 6579
 [771] 6580 6581 6614 6615 6616 6628 6643 6651 6655 6660 6675 6694 6695 6717
 [785] 6718 6741 6742 6743 6775 6776 6802 6805 6806 6811 6814 6819 6820 6854
 [799] 6857 6868 6882 6883 6884 6887 6890 6891 6895 6896 6900 6907 6931 6932
 [813] 6941 6942 6945 6949 6953 6958 6984 6985 6988 7005 7010 7036 7037 7058
 [827] 7061 7080 7083 7095 7102 7103 7106 7107 7108 7126 7144 7159 7164 7167
 [841] 7177 7180 7183 7187 7192 7193 7198 7221 7222 7223 7228 7229 7230 7236
 [855] 7239 7240 7245 7248 7279 7291 7294 7311 7312 7313 7320 7321 7324 7325
 [869] 7326 7327 7328 7329 7330 7346 7349 7350 7354 7355 7356 7421 7425 7426
 [883] 7427 7428 7454 7486 7508 7516 7536 7537 7540 7568 7573 7574 7575 7576
 [897] 7592 7593 7662 7663 7693 7722 7739 7740 7794 7795 7796 7803 7813 7843
 [911] 7849 7850 7851 7865 7889 7890 7891 7896 7916 7919 7920 7933 7936 7937
 [925] 7943 7961 7964 7965 7966 7982 8005 8006 8011 8055 8070 8075 8080 8088
 [939] 8099 8100 8105 8110 8125 8126 8127 8128 8129 8137 8140 8141 8153 8154
 [953] 8155 8158 8163 8164 8165 8166 8205 8206 8221 8226 8231 8238 8239 8240
 [967] 8241 8300 8301 8304 8333 8338 8358 8361 8364 8389 8397 8398 8399 8400
 [981] 8403 8406 8407 8410 8439 8440 8441 8444 8445 8452 8453 8460 8468 8473
 [995] 8474 8486 8487 8513 8514 8515 8536 8537 8553 8567 8568 8579 8580 8581
[1009] 8582 8585 8586 8587 8588 8617 8618 8629 8648 8651 8652 8665 8683 8688
[1023] 8699 8700 8701 8725 8726 8727 8736 8774 8779 8780 8784 8785 8786 8797
[1037] 8798 8799 8809 8810 8819 8845 8855 8872 8873 8874 8875 8883 8884 8885
[1051] 8886 8894 8898 8899 8907 8918 8954 8958 8959 8966 8975 8976 8998 9003
[1065] 9048 9049 9052 9063 9064 9065 9078 9086 9087 9112 9113 9137 9145 9174
[1079] 9178 9179 9180 9183 9186 9198 9199 9203 9204 9217 9224 9227 9228 9229
[1093] 9241 9242 9243 9255 9266 9267 9268 9269 9274 9279 9280 9331 9338 9339
[1107] 9371 9376 9381 9382 9383 9384 9385 9389 9390 9423 9428 9435 9443 9444
[1121] 9445 9446 9473 9474 9477 9505 9580 9588 9589 9592 9593 9594 9595 9612
[1135] 9617 9620 9672 9678 9681 9682 9704 9711 9712 9713 9803 9804 9807 9818
[1149] 9823 9824 9849 9879 9884 9885 9902 9910 9911 9939 9945 9946 9975 9979
[1163] 9980 9983 9986 9991

Part II. Debugging Techniques

Independent Reimplementation of Submodels

mydata[diff[1],1:4]
    V1       V2       V3       V4
197 99 98.58579 98.58579 98.58579
mydata[diff[1],c(5:8, 11)]
    V5 V6       V7 V8 V11
197 99 99 98.58579 99 100

Discuss: Given that V1-V8 are the neighbors elevation, and V11 is the elevation of the patch moved to, what does this tell you?

Answer: If at top of the hill, turtle stays put!

Part II. Debugging Techniques

Independent Reimplementation of Submodels

Discuss: Does this explain \(q\neq 0.475\) discrepancy from earlier?

Answer: Yes! If at top of hill, will not be equal to max of neighbors.

Part II. Debugging Techniques

Independent Reimplementation of Submodels

From ODD Submodels:

“..to ‘move uphill’ is defined specifically as moving to the neighbor patch that has the highest elevation; if two patches have the same elevation, one is chosen randomly. ‘Move randomly’ is defined as moving to one of the neighboring patches, with equal probability of choosing any patch. ‘Neighbor patches’ are the eight patches surrounding the butterfly’s current patch.”

OOPS!

Part II. Debugging Techniques

Independent Reimplementation of Submodels

doh

Part II. Debugging Techniques

Independent Reimplementation of Submodels

How to fix? Change

[uphill elevation]

to

[move-to max-one-of neighbors [elevation]]

This was a “Misunderstanding Primitives” error!!!

Intro to Quantitative Biology, Spring 2023