Testing Your Program

• Learn the importance of version control.
• Understand the concept that using ABMs for science means:
  1. measuring system response and behavior through producing quantitative output, and
  2. conduct simulation experiments (in silico).
• Perform your first simulation experiment.
• Define and initialize a global variable by creating a slider or switch.
• Develop an understanding of what reporters are and how to write them.

• Start learning how to find and fix mistakes in your code.
• Learn to create output by:
  1. writing text to an output window,
  2. creating a time-series plot, and
  3. exporting plot results to a file.
• Create a version of the Butterfly model that uses real topographic data by importing data into NetLogo.

• GUI (Graphical User Interface) elements on the Interface both define and initialize a global variable. What this means if you move a variable to the Interface:
  • You should comment out the variable in the globals [ ] chunk of code. I would also put an additional comment at the end stating it is on the Interface. For example:
    

globals
[
  ; q  ; (Interface Slider) Probability that ...
]

• GUI (Graphical User Interface) elements on the Interface both define and initialize a global variable. What this means if you move a variable to the Interface:
  • You should remove or comment out the variable initialization in the setup procedure. If you choose to comment and not remove, I would also put the same comment from the globals chunk in here as well. For example:

to setup
[
  ; set q 0.2  ; (Interface Slider)
]

• New variables have a default value of zero until assigned another value.
• Good practice is to explicitly set it to zero anyways in the code.

Tips to reduce probability of getting stuck and increase probability of getting unstuck.

• Proceed slowly! Add small chunks of code at a time, check for syntax (using Check button) and runtime (using Go button) errors, and verify and validate code (i.e. is it doing what it is supposed to do?) For verification and validation:
  1. Create a Step button that steps through the program one tick at a time.
  2. Use show statements throughout your code.
  3. Use Agent Monitors.

Tips to reduce probability of getting stuck and increase probability of getting unstuck.

• Think logically!
  
  1. “It should work this way because…”, i.e. critically assess output and results and see if they make sense
  2. “Let me try this and see what happens…”

“The productive modeler simply assumes that software mistakes are inevitable and continually searches for them.”
- Railsback & Grimm

• Understand the difference between validation and verification.
• Seven common kinds of software errors that will blow your mind!
• Ten important techniques for finding and fixing software errors everyone should know.
• Understand why and how to document software tests.

• Typographical Errors
• Syntax Errors
• Misunderstanding Primitives
• Wrong Display Settings: Use resize-world in setup!
• Logic Errors: Runs but results incorrect
• Run-time Errors: No syntax or logic errors, but breaks on Go (sometimes)
• Formulation Errors: Incorrect assumptions & model decisions

Syntax Checking: Chunk-it and use skeleton code!

ifelse (xcor >= min-marriage-age)
[ show "If" ]
[ show "Else" ]

ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ show "If" ]
[ show "Else" ]

ifelse (xcor >= min-marriage-age) and
       (random-float 1.0 < 0.1)
[ set married? true ]
[ show "Else" ]

Visual Testing: Use visual cues of variables!

• Use scale-color to color turtles or patches based on their variables.
• Use label and plabel to check turtle or patch information.
• Use Agent and Patch Monitor.
• Use a smaller World to test things (actually, testing on smaller problem is a more general technique)
• Slow down the simulation and/or use a step button.

Print Statements

• For procedures:

to dostuff
  show "Starting procedure X"
  ; Do stuff
  show "Ending procedure X"
end

• For variables:

observer> show word "num turtles = " count turtles
observer: "num turtles = 0"

Code Reviews

• Reviewer's job:
  • Verification: Does the code match the ODD model formulation?
  • Fresh set of eyes can more easily find bugs (sometimes).
  • Make sure code is well-organized and easy to understand.
  • Write code as if someone will eventually read AND USE IT, even if you do not plan on it being used.

Statistical Analysis of File Output

Question: Is the probability butterflies move to the highest neighbor patch really \( q \)?

Answer: No. It is the approximate proportion \[ q + \frac{1-q}{8}. \]

For \( q = 0.4 \), we would expect the butterfly to move to the highest neighbor patch with probability 0.475.

Statistical Analysis of File Output

file-type
file-print
file-open
file-close

Statistical Analysis of File Output

mydata <- read.csv("TestOutput.csv", header=FALSE)
str(mydata)

'data.frame':   10000 obs. of  9 variables:
 $ V1: num  14.8 16.8 16.9 16.9 17.9 ...
 $ V2: num  15.6 14.9 17.9 18.9 19.9 ...
 $ V3: num  15.5 14.8 16.8 17.9 18.9 ...
 $ V4: num  15.8 14.9 15.9 16.9 18.7 ...
 $ V5: num  15.5 16.6 17.9 19 17.8 ...
 $ V6: num  16.3 15.9 15.8 17 19.9 ...
 $ V7: num  14.7 15.6 17.8 18 19.7 ...
 $ V8: num  15.6 16.9 15.9 18.9 17.9 ...
 $ V9: num  15.8 16.9 17.9 18.9 19.9 ...

Statistical Analysis of File Output

moved.to.highest <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,9]})

moved.to.highest <- as.integer(moved.to.highest)

Statistical Analysis of File Output

prop.test(sum(moved.to.highest), 10000, p = 0.4)

    1-sample proportions test with continuity correction

data:  sum(moved.to.highest) out of 10000, null probability 0.4
X-squared = 14.805, df = 1, p-value = 0.0001192
alternative hypothesis: true p is not equal to 0.4
95 percent confidence interval:
 0.4092131 0.4286495
sample estimates:
     p 
0.4189 

Statistical Analysis of File Output

Discuss: Wait, what?!? Result doesn’t even contain 0.475 in the confidence interval. Explain why the estimate is smaller than 0.475.

Hint: What if you are at the top of a hill?

Regardless, something is wrong. We are having a problem with verification, i.e. the program is not doing what it is supposed to do.

Independent Reimplementation of Submodels

Test movement submodel.

mydata <- read.csv("SubmodelOutput.csv", header=FALSE)
str(mydata)

Independent Reimplementation of Submodels

Test movement submodel.

'data.frame':   10000 obs. of  11 variables:
 $ V1 : num  15.5 15.6 17.6 15.7 17.1 ...
 $ V2 : num  15.8 16.4 15.8 17.2 16.5 ...
 $ V3 : num  15.6 16.6 15.6 17.4 16.5 ...
 $ V4 : num  15.5 16.5 15.5 16.6 15.5 ...
 $ V5 : num  16.3 14.9 16.5 15.6 15.6 ...
 $ V6 : num  15.6 15.7 17.8 17.6 16.3 ...
 $ V7 : num  14.8 15.6 17.4 16.3 16.4 ...
 $ V8 : num  14.7 16.3 16.8 15.5 16 ...
 $ V9 : num  0.794 0.225 0.956 0.844 0.556 ...
 $ V10: num  0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 ...
 $ V11: num  15.5 16.6 16.5 15.7 15.5 ...

Independent Reimplementation of Submodels

moved.up <- sapply(1:10000, function (x) {max(mydata[x,1:8]) == mydata[x,11]})

should.moved.up <- (mydata[,9] < mydata[,10])

(diff <- which((moved.up == FALSE) & (should.moved.up == TRUE)))

   [1]   92   98  115  116  121  124  125  126  127  136  147  156  157  165
  [15]  168  172  204  221  224  232  233  234  237  251  289  364  365  373
  [29]  378  381  384  387  438  445  448  462  470  486  487  488  489  490
  [43]  515  516  522  525  541  542  545  548  557  567  570  592  608  611
  [57]  644  647  652  671  721  722  723  724  769  814  830  840  849  850
  [71]  855  856  874  875  883  884  885  908  931  958  964  965  966  971
  [85]  978  981  992 1011 1030 1047 1060 1061 1108 1109 1129 1132 1133 1143
  [99] 1167 1210 1258 1261 1273 1286 1299 1310 1311 1312 1317 1318 1323 1324
 [113] 1365 1366 1411 1412 1413 1440 1467 1470 1483 1484 1485 1486 1487 1492
 [127] 1495 1505 1506 1507 1517 1520 1521 1529 1532 1533 1536 1552 1555 1561
 [141] 1562 1612 1617 1652 1699 1710 1719 1724 1727 1747 1750 1753 1754 1768
 [155] 1769 1770 1775 1776 1777 1778 1804 1809 1810 1811 1830 1831 1836 1855
 [169] 1860 1898 1907 1917 1927 1928 1935 1940 1943 1946 1947 1948 1949 1950
 [183] 1959 1960 1965 1979 1980 1981 2009 2016 2020 2021 2030 2033 2038 2053
 [197] 2056 2059 2060 2061 2076 2096 2097 2098 2117 2118 2121 2135 2136 2137
 [211] 2138 2139 2170 2176 2180 2198 2205 2213 2236 2237 2250 2263 2266 2267
 [225] 2273 2278 2279 2290 2291 2294 2295 2317 2352 2373 2374 2387 2394 2412
 [239] 2413 2419 2440 2441 2442 2451 2461 2479 2503 2508 2512 2517 2518 2519
 [253] 2520 2521 2536 2545 2551 2552 2553 2558 2565 2578 2618 2619 2620 2621
 [267] 2624 2625 2644 2661 2700 2701 2712 2715 2716 2727 2730 2735 2745 2748
 [281] 2751 2752 2753 2765 2867 2890 2903 2906 2927 2942 2968 2973 2974 2975
 [295] 2976 2977 2985 2986 2987 2988 3000 3001 3014 3037 3042 3043 3046 3069
 [309] 3105 3111 3112 3113 3116 3119 3130 3131 3140 3141 3142 3143 3144 3145
 [323] 3146 3147 3148 3159 3160 3161 3162 3163 3164 3165 3185 3204 3205 3206
 [337] 3207 3214 3215 3216 3217 3220 3221 3253 3254 3281 3282 3285 3304 3305
 [351] 3313 3333 3394 3395 3411 3434 3445 3452 3458 3469 3470 3473 3486 3496
 [365] 3497 3498 3507 3508 3549 3554 3563 3567 3584 3585 3595 3613 3646 3647
 [379] 3648 3652 3653 3656 3657 3658 3661 3670 3701 3702 3708 3709 3712 3713
 [393] 3714 3758 3769 3774 3775 3806 3809 3812 3851 3861 3862 3867 3868 3877
 [407] 3878 3886 3887 3888 3889 3902 3971 3976 3994 3999 4000 4018 4019 4033
 [421] 4054 4061 4075 4096 4103 4104 4110 4111 4112 4113 4118 4128 4129 4130
 [435] 4135 4136 4149 4157 4158 4159 4177 4186 4187 4193 4194 4201 4202 4232
 [449] 4247 4248 4249 4252 4275 4283 4288 4289 4297 4298 4299 4300 4318 4327
 [463] 4330 4333 4334 4338 4343 4367 4370 4371 4378 4391 4404 4405 4406 4407
 [477] 4408 4417 4418 4440 4445 4463 4469 4470 4471 4474 4475 4476 4479 4543
 [491] 4548 4556 4564 4565 4590 4615 4616 4619 4631 4636 4637 4640 4641 4658
 [505] 4666 4671 4674 4677 4688 4706 4707 4729 4759 4770 4771 4782 4783 4790
 [519] 4798 4807 4810 4819 4822 4834 4835 4836 4837 4838 4871 4889 4892 4893
 [533] 4894 4927 4947 4948 4969 4970 4993 4994 4995 5019 5022 5050 5067 5068
 [547] 5071 5072 5084 5087 5107 5126 5127 5128 5129 5167 5168 5169 5172 5187
 [561] 5196 5199 5202 5208 5209 5210 5211 5227 5230 5236 5250 5251 5252 5253
 [575] 5264 5265 5266 5271 5272 5296 5297 5346 5350 5351 5354 5355 5375 5376
 [589] 5377 5383 5384 5385 5386 5387 5391 5394 5412 5413 5414 5415 5416 5430
 [603] 5440 5441 5442 5453 5457 5458 5459 5460 5461 5532 5533 5552 5555 5556
 [617] 5559 5566 5567 5568 5569 5592 5593 5596 5597 5600 5614 5622 5627 5678
 [631] 5686 5703 5706 5714 5724 5727 5728 5731 5732 5733 5734 5735 5738 5748
 [645] 5771 5772 5773 5784 5785 5795 5803 5810 5840 5870 5876 5877 5878 5887
 [659] 5888 5899 5900 5911 5919 5923 5934 5941 5962 5963 5975 5985 5999 6000
 [673] 6009 6055 6064 6071 6072 6073 6076 6077 6087 6117 6124 6125 6126 6131
 [687] 6146 6147 6148 6151 6154 6161 6175 6185 6249 6250 6251 6274 6277 6278
 [701] 6293 6294 6295 6296 6329 6336 6343 6373 6379 6382 6383 6386 6387 6388
 [715] 6389 6390 6391 6392 6412 6417 6418 6434 6441 6447 6448 6451 6452 6455
 [729] 6462 6492 6493 6496 6497 6500 6510 6530 6540 6555 6556 6557 6566 6572
 [743] 6575 6576 6577 6580 6597 6598 6624 6632 6650 6735 6742 6743 6744 6745
 [757] 6746 6759 6765 6766 6771 6774 6781 6812 6832 6835 6836 6840 6841 6844
 [771] 6851 6855 6856 6902 6918 6919 6925 6936 6937 6938 6960 6963 6970 6971
 [785] 7014 7028 7052 7053 7054 7055 7058 7059 7060 7064 7065 7086 7087 7102
 [799] 7106 7107 7108 7113 7123 7124 7127 7161 7162 7174 7181 7189 7218 7223
 [813] 7224 7233 7244 7252 7253 7282 7283 7289 7317 7318 7319 7320 7331 7332
 [827] 7333 7340 7363 7364 7365 7366 7371 7381 7413 7414 7417 7443 7452 7453
 [841] 7454 7455 7458 7459 7489 7493 7500 7527 7530 7531 7549 7550 7551 7565
 [855] 7594 7601 7602 7605 7606 7607 7626 7641 7647 7648 7657 7658 7659 7674
 [869] 7675 7686 7713 7718 7737 7752 7753 7756 7757 7784 7817 7818 7832 7840
 [883] 7852 7853 7854 7855 7865 7877 7878 7882 7883 7891 7915 7928 7933 7934
 [897] 7973 7974 7981 7984 7987 8005 8006 8007 8008 8011 8041 8048 8049 8050
 [911] 8054 8055 8056 8094 8097 8100 8112 8123 8128 8150 8189 8192 8193 8194
 [925] 8197 8198 8199 8227 8228 8231 8239 8240 8241 8253 8256 8257 8289 8292
 [939] 8300 8301 8302 8303 8315 8316 8319 8320 8330 8346 8358 8359 8377 8393
 [953] 8394 8400 8416 8419 8428 8435 8436 8460 8470 8489 8490 8519 8527 8528
 [967] 8531 8549 8550 8554 8558 8583 8586 8590 8591 8594 8597 8598 8605 8630
 [981] 8641 8642 8643 8652 8653 8686 8695 8701 8742 8745 8746 8747 8771 8784
 [995] 8787 8802 8803 8804 8809 8810 8815 8816 8825 8839 8840 8841 8842 8843
[1009] 8844 8850 8861 8875 8893 8898 8899 8905 8910 8915 8922 8923 8924 8927
[1023] 8965 8966 8972 8973 9010 9016 9019 9020 9021 9022 9023 9024 9025 9046
[1037] 9053 9054 9057 9060 9068 9071 9072 9073 9074 9086 9094 9095 9098 9134
[1051] 9135 9138 9151 9152 9153 9154 9155 9156 9190 9191 9192 9193 9194 9195
[1065] 9201 9202 9207 9208 9213 9216 9217 9218 9223 9224 9225 9260 9263 9275
[1079] 9280 9309 9312 9313 9316 9319 9320 9327 9328 9329 9330 9337 9338 9351
[1093] 9354 9355 9356 9357 9372 9379 9380 9381 9382 9383 9384 9387 9388 9389
[1107] 9390 9400 9418 9419 9424 9425 9448 9453 9456 9474 9490 9491 9495 9502
[1121] 9518 9526 9529 9530 9566 9567 9568 9588 9639 9640 9641 9652 9686 9689
[1135] 9690 9702 9707 9714 9715 9733 9760 9763 9764 9769 9774 9779 9792 9815
[1149] 9822 9823 9829 9832 9833 9839 9840 9845 9846 9847 9850 9868 9869 9874
[1163] 9875 9960 9961 9996

Independent Reimplementation of Submodels

         V1 V2       V3 V4
92 48.58579 49 48.58579 49

         V5 V6 V7       V8 V11
92 48.58579 49 49 48.58579  50

Discuss: Given that V1-V8 are the neighbors elevation, and V11 is the elevation of the patch moved to, what does this tell you?

Answer: If at top of the hill, turtle stays put!

Independent Reimplementation of Submodels

Discuss: Does this explain \( q\neq 0.475 \) discrepancy from earlier?

Answer: Yes! If at top of hill, will not be equal to max of neighbors.

Independent Reimplementation of Submodels

From ODD Submodels:

“..to 'move uphill' is defined specifically as moving to the neighbor patch that has the highest elevation; if two patches have the same elevation, one is chosen randomly. 'Move randomly' is defined as moving to one of the neighboring patches, with equal probability of choosing any patch. 'Neighbor patches' are the eight patches surrounding the butterfly's current patch.”

OOPS!

Independent Reimplementation of Submodels

How to fix? Change

[uphill elevation]

to

[move-to max-one-of neighbors [elevation]]

This was a “Misunderstanding Primitives” error!!!