Testing Your Program

• Learn the importance of version control.
• Understand the concept that using ABMs for science means:
  1. measuring system response and behavior through producing quantitative output, and
  2. conduct simulation experiments (in silico).
• Perform your first simulation experiment.
• Define and initialize a global variable by creating a slider or switch.
• Develop an understanding of what reporters are and how to write them.

• Start learning how to find and fix mistakes in your code.
• Learn to create output by:
  1. writing text to an output window,
  2. creating a time-series plot, and
  3. exporting plot results to a file.
• Create a version of the Butterfly model that uses real topographic data by importing data into NetLogo.

• GUI (Graphical User Interface) elements on the Interface both define and initialize a global variable. What this means if you move a variable to the Interface:
  • You should comment out the variable in the globals [ ] chunk of code. I would also put an additional comment at the end stating it is on the Interface. For example:
    

globals
[
  ; q  ; (Interface Slider) Probability that ...
]

• GUI (Graphical User Interface) elements on the Interface both define and initialize a global variable. What this means if you move a variable to the Interface:
  • You should remove or comment out the variable initialization in the setup procedure. If you choose to comment and not remove, I would also put the same comment from the globals chunk in here as well. For example:

to setup
[
  ; set q 0.2  ; (Interface Slider)
]

• New variables have a default value of zero until assigned another value.
• Good practice is to explicitly set it to zero anyways in the code.

Tips to reduce probability of getting stuck and increase probability of getting unstuck.

• Proceed slowly! Add small chunks of code at a time, check for syntax (using Check button) and runtime (using Go button) errors, and verify and validate code (i.e. is it doing what it is supposed to do?) For verification and validation:
  1. Create a Step button that steps through the program one tick at a time.
  2. Use show statements throughout your code.
  3. Use Agent Monitors.

Tips to reduce probability of getting stuck and increase probability of getting unstuck.

• Think logically!
  
  1. “It should work this way because…”, i.e. critically assess output and results and see if they make sense
  2. “Let me try this and see what happens…”

“The productive modeler simply assumes that software mistakes are inevitable and continually searches for them.”
- Railsback & Grimm

• Understand the difference between validation and verification.
• Seven common kinds of software errors that will blow your mind!
• Ten important techniques for finding and fixing software errors everyone should know.
• Understand why and how to document software tests.

• Typographical Errors
• Syntax Errors
• Misunderstanding Primitives
• Wrong Display Settings: Use resize-world in setup!
• Logic Errors: Runs but results incorrect
• Run-time Errors: No syntax or logic errors, but breaks on Go (sometimes)
• Formulation Errors: Incorrect assumptions & model decisions

Syntax Checking: Chunk-it and use skeleton code!

ifelse (xcor >= min-marriage-age)
[ show "If" ]
[ show "Else" ]

ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ show "If" ]
[ show "Else" ]

ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ set married? true ]
[ show "Else" ]

Visual Testing: Use visual cues of variables!

• Use scale-color to color turtles or patches based on their variables.
• Use label and plabel to check turtle or patch information.
• Use Agent and Patch Monitor.
• Use a smaller World to test things (actually, testing on smaller problem is a more general technique)
• Slow down the simulation and/or use a step button.

Print Statements

• For procedures:

to dostuff
  show "Starting procedure X"
  ; Do stuff
  show "Ending procedure X"
end

• For variables:

observer> show word "num turtles = " count turtles
observer: "num turtles = 0"

Code Reviews

• Reviewer's job:
  • Verification: Does the code match the ODD model formulation?
  • Fresh set of eyes can more easily find bugs (sometimes).
  • Make sure code is well-organized and easy to understand.
  • Write code as if someone will eventually read AND USE IT, even if you do not plan on it being used.

Statistical Analysis of File Output

Question: Is the probability butterflies move to the highest neighbor patch really \( q \)?

Answer: No. It is the approximate proportion \[ q + \frac{1-q}{8}. \]

For \( q = 0.4 \), we would expect the butterfly to move to the highest neighbor patch with probability 0.475.

Statistical Analysis of File Output

file-type
file-print
file-open
file-close

Statistical Analysis of File Output

mydata <- read.csv("TestOutput.csv", header=FALSE)
str(mydata)

'data.frame':   10000 obs. of  9 variables:
 $ V1: num  14.8 16.8 16.9 16.9 17.9 ...
 $ V2: num  15.6 14.9 17.9 18.9 19.9 ...
 $ V3: num  15.5 14.8 16.8 17.9 18.9 ...
 $ V4: num  15.8 14.9 15.9 16.9 18.7 ...
 $ V5: num  15.5 16.6 17.9 19 17.8 ...
 $ V6: num  16.3 15.9 15.8 17 19.9 ...
 $ V7: num  14.7 15.6 17.8 18 19.7 ...
 $ V8: num  15.6 16.9 15.9 18.9 17.9 ...
 $ V9: num  15.8 16.9 17.9 18.9 19.9 ...

Statistical Analysis of File Output

moved.to.highest <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,9]})

moved.to.highest <- as.integer(moved.to.highest)

Statistical Analysis of File Output

prop.test(sum(moved.to.highest), 10000, p = 0.4)

    1-sample proportions test with continuity correction

data:  sum(moved.to.highest) out of 10000, null probability 0.4
X-squared = 14.805, df = 1, p-value = 0.0001192
alternative hypothesis: true p is not equal to 0.4
95 percent confidence interval:
 0.4092131 0.4286495
sample estimates:
     p 
0.4189 

Statistical Analysis of File Output

Discuss: Wait, what?!? Result doesn’t even contain 0.475 in the confidence interval. Explain why the estimate is smaller than 0.475.

Hint: What if you are at the top of a hill?

Regardless, something is wrong. We are having a problem with verification, i.e. the program is not doing what it is supposed to do.

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)

Independent Reimplementation of Submodels

Test movement submodel.

'data.frame':   10000 obs. of  11 variables:
 $ V1 : num  15.6 17 18.3 18.4 20.5 ...
 $ V2 : num  15.5 16.1 17.8 17.7 19.1 ...
 $ V3 : num  14.7 15.6 17.5 20.5 21.1 ...
 $ V4 : num  15.5 16.4 19.1 18.3 18.3 ...
 $ V5 : num  14.8 14.9 16.3 19.7 19.8 ...
 $ V6 : num  16.3 15.5 16.9 19.8 20.4 ...
 $ V7 : num  15.6 16.9 17 18.9 19.6 ...
 $ V8 : num  15.8 17.7 18.4 19.2 18.9 ...
 $ V9 : num  0.032 0.156 0.405 0.422 0.82 ...
 $ V10: num  0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 ...
 $ V11: num  16.3 17.7 19.1 19.7 20.5 ...

Independent Reimplementation of Submodels

moved.up <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,11]})

should.moved.up <- (mydata[,9] < mydata[,10])

(diff <- which((moved.up == FALSE) & (should.moved.up == TRUE)))

   [1]  163  177  194  197  208  213  220  221  240  243  250  255  266  270
  [15]  284  287  288  298  299  310  313  316  329  330  333  334  337  345
  [29]  346  347  354  364  367  375  376  379  384  413  423  424  433  434
  [43]  435  440  454  457  458  466  480  481  484  489  505  506  552  553
  [57]  565  566  579  587  590  591  594  604  626  627  632  664  680  688
  [71]  689  690  697  704  727  728  749  769  794  799  800  801  810  811
  [85]  816  823  824  835  849  869  870  871  872  873  874  880  899  900
  [99]  901  902  903  921  922  938  946  947  964  971  975  978  992  993
 [113] 1000 1019 1020 1021 1026 1027 1028 1029 1033 1038 1055 1094 1095 1098
 [127] 1103 1104 1107 1127 1128 1139 1140 1145 1148 1165 1177 1182 1186 1189
 [141] 1190 1191 1206 1207 1208 1213 1216 1217 1218 1223 1231 1234 1262 1263
 [155] 1287 1295 1331 1334 1340 1347 1348 1351 1358 1363 1364 1365 1394 1408
 [169] 1409 1410 1413 1431 1440 1456 1528 1529 1534 1541 1561 1562 1563 1564
 [183] 1569 1570 1571 1572 1573 1574 1575 1582 1588 1594 1600 1614 1634 1637
 [197] 1672 1681 1698 1783 1822 1843 1844 1845 1851 1854 1867 1868 1869 1870
 [211] 1871 1895 1901 1923 1926 1942 1949 1950 1957 1958 1959 1960 1963 1964
 [225] 1969 1988 1989 2006 2007 2035 2038 2039 2048 2049 2057 2066 2074 2075
 [239] 2093 2103 2137 2140 2143 2147 2158 2175 2176 2179 2182 2183 2194 2215
 [253] 2224 2249 2254 2255 2269 2277 2286 2299 2300 2301 2302 2307 2308 2309
 [267] 2310 2311 2335 2346 2357 2372 2406 2407 2435 2438 2449 2450 2455 2456
 [281] 2457 2458 2459 2462 2472 2473 2474 2475 2481 2493 2494 2495 2496 2497
 [295] 2507 2508 2536 2579 2582 2583 2586 2593 2598 2601 2614 2617 2618 2619
 [309] 2652 2653 2654 2668 2669 2670 2671 2672 2673 2676 2692 2693 2701 2702
 [323] 2715 2716 2717 2718 2724 2727 2735 2746 2747 2757 2772 2775 2787 2790
 [337] 2814 2815 2820 2821 2822 2846 2851 2864 2875 2878 2879 2880 2896 2922
 [351] 2973 2974 2980 2983 2984 2985 2986 3005 3006 3023 3029 3090 3093 3113
 [365] 3114 3150 3171 3172 3191 3192 3200 3215 3241 3247 3248 3249 3273 3284
 [379] 3285 3286 3308 3340 3343 3350 3361 3365 3395 3396 3399 3410 3415 3416
 [393] 3422 3423 3465 3466 3467 3468 3490 3491 3492 3495 3501 3502 3503 3508
 [407] 3509 3510 3513 3523 3531 3534 3543 3544 3553 3596 3604 3633 3634 3678
 [421] 3679 3680 3686 3687 3688 3693 3698 3699 3700 3717 3718 3719 3725 3726
 [435] 3727 3728 3734 3739 3750 3753 3758 3759 3760 3761 3762 3763 3766 3767
 [449] 3768 3769 3773 3774 3777 3778 3779 3780 3790 3791 3796 3797 3814 3826
 [463] 3831 3836 3837 3838 3839 3842 3867 3870 3889 3908 3909 3919 3932 3933
 [477] 3951 3952 3953 3958 3959 3960 4022 4025 4026 4073 4078 4081 4082 4086
 [491] 4087 4098 4103 4113 4159 4164 4180 4191 4198 4201 4202 4213 4216 4217
 [505] 4218 4219 4220 4236 4244 4270 4271 4293 4294 4295 4311 4343 4344 4347
 [519] 4351 4366 4367 4382 4398 4399 4432 4448 4449 4450 4464 4465 4466 4478
 [533] 4479 4509 4526 4529 4556 4557 4568 4582 4583 4584 4585 4599 4618 4628
 [547] 4639 4643 4654 4660 4661 4662 4721 4730 4758 4759 4760 4772 4773 4774
 [561] 4775 4801 4822 4823 4835 4841 4847 4848 4861 4864 4865 4868 4871 4930
 [575] 4931 4934 4945 4955 4956 4969 4972 4973 4995 5013 5019 5031 5067 5072
 [589] 5073 5090 5115 5116 5117 5150 5167 5171 5185 5254 5257 5258 5259 5289
 [603] 5290 5295 5299 5304 5315 5337 5340 5341 5342 5358 5359 5360 5369 5378
 [617] 5379 5383 5386 5398 5399 5400 5401 5402 5408 5409 5410 5411 5412 5419
 [631] 5420 5421 5422 5435 5436 5485 5486 5503 5504 5520 5525 5541 5549 5550
 [645] 5551 5552 5555 5586 5591 5594 5598 5601 5610 5611 5612 5613 5614 5623
 [659] 5626 5642 5643 5646 5647 5650 5659 5681 5684 5687 5688 5689 5725 5726
 [673] 5727 5736 5757 5788 5802 5803 5804 5816 5817 5818 5819 5832 5835 5849
 [687] 5850 5864 5865 5868 5869 5870 5871 5880 5897 5900 5908 5935 5938 5939
 [701] 5942 5945 5946 5947 5957 5980 5983 5990 5991 5994 5995 5996 5999 6000
 [715] 6008 6009 6010 6030 6031 6050 6051 6052 6053 6054 6055 6056 6057 6060
 [729] 6061 6075 6080 6087 6110 6113 6114 6115 6150 6151 6172 6173 6199 6200
 [743] 6217 6230 6233 6234 6235 6236 6237 6238 6239 6242 6249 6250 6251 6252
 [757] 6298 6327 6328 6329 6349 6350 6351 6359 6373 6378 6390 6436 6451 6452
 [771] 6455 6456 6469 6470 6473 6476 6477 6478 6479 6482 6508 6509 6512 6515
 [785] 6516 6520 6527 6530 6531 6532 6553 6563 6564 6573 6584 6598 6609 6610
 [799] 6614 6615 6618 6619 6622 6631 6634 6635 6638 6645 6685 6703 6704 6717
 [813] 6729 6735 6739 6742 6747 6759 6760 6763 6764 6779 6782 6783 6784 6785
 [827] 6792 6812 6815 6820 6821 6822 6832 6847 6848 6851 6854 6855 6856 6867
 [841] 6877 6878 6894 6921 6929 6930 6935 6936 6945 6946 6949 6952 6953 6954
 [855] 6955 6974 6975 6985 6994 7046 7062 7084 7085 7103 7104 7107 7154 7167
 [869] 7177 7178 7179 7180 7181 7189 7204 7210 7213 7214 7217 7218 7219 7222
 [883] 7225 7226 7227 7230 7238 7247 7252 7255 7293 7329 7330 7331 7337 7338
 [897] 7358 7365 7366 7367 7372 7378 7379 7387 7468 7469 7503 7504 7507 7510
 [911] 7511 7518 7529 7530 7531 7559 7572 7575 7586 7624 7667 7672 7687 7690
 [925] 7695 7717 7718 7755 7762 7782 7785 7786 7787 7788 7789 7790 7796 7801
 [939] 7822 7832 7835 7836 7837 7845 7853 7854 7869 7873 7874 7879 7891 7892
 [953] 7897 7902 7907 7908 7909 7932 7933 7937 7938 7960 7963 7964 7970 7975
 [967] 7985 7994 8004 8007 8021 8022 8041 8046 8058 8059 8088 8089 8111 8112
 [981] 8125 8126 8132 8133 8138 8154 8160 8175 8196 8207 8224 8263 8264 8271
 [995] 8276 8281 8293 8294 8295 8296 8315 8326 8333 8336 8339 8350 8362 8363
[1009] 8364 8375 8376 8377 8378 8396 8424 8425 8445 8460 8478 8481 8482 8491
[1023] 8496 8504 8578 8585 8586 8587 8588 8589 8590 8591 8592 8598 8607 8608
[1037] 8611 8618 8626 8631 8641 8642 8643 8644 8645 8646 8647 8648 8653 8656
[1051] 8657 8660 8665 8670 8671 8672 8685 8686 8766 8778 8785 8790 8796 8797
[1065] 8798 8804 8805 8806 8820 8830 8872 8879 8884 8885 8890 8897 8898 8899
[1079] 8905 8906 8924 8927 8930 8938 8953 8954 8958 8963 8964 8970 8979 8982
[1093] 8998 9005 9013 9014 9015 9016 9026 9027 9030 9067 9080 9081 9091 9092
[1107] 9093 9137 9138 9139 9142 9143 9150 9160 9161 9162 9163 9166 9192 9193
[1121] 9228 9243 9274 9283 9284 9285 9304 9312 9332 9335 9340 9343 9370 9382
[1135] 9391 9392 9397 9398 9399 9415 9416 9417 9418 9419 9431 9453 9462 9463
[1149] 9468 9481 9486 9487 9508 9509 9515 9516 9524 9528 9535 9536 9589 9613
[1163] 9614 9615 9616 9645 9652 9669 9699 9700 9708 9713 9714 9715 9730 9742
[1177] 9769 9770 9771 9798 9809 9810 9819 9822 9837 9840 9841 9845 9850 9851
[1191] 9854 9855 9868 9869 9882 9885 9908 9916 9917 9920 9975 9978 9979 9982
[1205] 9987 9995

Independent Reimplementation of Submodels

    V1       V2 V3       V4
163 99 98.58579 99 98.58579

          V5 V6 V7       V8 V11
163 98.58579 99 99 98.58579 100

Discuss: Given that V1-V8 are the neighbors elevation, and V11 is the elevation of the patch moved to, what does this tell you?

Answer: If at top of the hill, turtle stays put!

Independent Reimplementation of Submodels

Discuss: Does this explain \( q\neq 0.475 \) discrepancy from earlier?

Answer: Yes! If at top of hill, will not be equal to max of neighbors.

Independent Reimplementation of Submodels

From ODD Submodels:

“..to 'move uphill' is defined specifically as moving to the neighbor patch that has the highest elevation; if two patches have the same elevation, one is chosen randomly. 'Move randomly' is defined as moving to one of the neighboring patches, with equal probability of choosing any patch. 'Neighbor patches' are the eight patches surrounding the butterfly's current patch.”

OOPS!

Independent Reimplementation of Submodels

How to fix? Change

[uphill elevation]

to

[move-to max-one-of neighbors [elevation]]

This was a “Misunderstanding Primitives” error!!!