This document is a tutorial on how to calculate Goodman-Kruskal gamma correlations in R and RStudio. These calculations are important for metacognitive research in which judgments of learning (JOLs), feelings of knowing (FOKs), and confidence judgments (CJs) are compared to memory performance to assess relative accuracy.1

What are Gamma Correlations?

Simply put, gamma correlations are a measure of ordinal association between two vectors of numbers. Instead of measuring the relationship between two sets of continuous or interval data, such as what a “normal” Pearson correlation does, the gamma correlation translates scale numbers into ordinal (“rank”) data. Thus, scale metacognitive judgments, such as JOLs ranging from 0 - 100, are not analyzed by their absolute values; rather, they are considered rank judgments of memory likelihood. A judgment of “10”, for example, would be considered a greater percieved likelihood than a judgment of “1”, but would not be considered 10 times more likely than “1” in a gamma correlation.

Gamma correlations are computed pair-wise, meaning that the memory outcomes (0 or 1) are paired with a corresponding judgment (JOL, FOK, etc.) and compared to all other pairs of numbers. All pairs that are ranked on the same order for both variables are considered “concordant”, and all pairs that are ranked on the reverse order are considered “disconcordant”. Instances are tallied, and the difference between the concordant and disconcordant pairs is divided by the sum of the concordant and disconcordant pairs to return a vector from -1 to 1, with values less than 0 indicating a negative relationship between the judgment and subsequent memory (e.g. larger JOLs for items that are unrecalled) and values greater than 0 indicating a positive relationship (e.g. larger JOLs for items that are recalled).

\(G = \frac{N_c - N_d}{N_c + N_d}\)

Calculations in R

As a note: If you are viewing this in R as a Notebook, then you can execute the code in the boxes by clicking Run or by clicking inside of the chunk and pressing Ctrl+Shift+Enter. If you are viewing this as a PDF or HTML, the output will appear below the associated box.

The first thing that you will need to do (assuming that you already have RStudio set up) is install and initialize the packages associated with gammas and the computations below. The code here will only initialize it, but installation code is available in the commented sections.

#install.packages(Hmisc)
require(Hmisc)

Hmisc is a freely-available package (abbreviated from “Harrell Miscellaneous”) that supplements the standard R statistics repository. Most of the functions that are available here are of no interest to us, except for rcorr.cens, which provides the Goodman-Kruskal correlation – provided that you use the correct code. We will get into that later.

To begin, let’s generate a dataset to work with. For this example, we will be using scale FOKs from 0 to 100, with an FOK of 0 meaning no confidence and an FOK of 100 meaning absolute confidence, and recognition outcome data, with 0 meaning an incorrect recognition and 1 meaning a correct recognition. The code below randomly generates 60 FOKs from 0 - 100 and 60 recognition memory outcomes:

# Randomly generate FOk dataset
recog <- sample(0:1,60,replace=TRUE)
fok <- sample(0:100,60,replace=TRUE)
# Show generated data
print(recog)
 [1] 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 1 1 1 1 0 1 1 1 1 0 1 0 1 1 1 1 0 0 0 1 0 1 1 0 0 1 0 1 1 1 0 0 1 0 1 1 1 1 1 0 1 0 0 1 0
print(fok)
 [1] 13 29 12 88 74 71 56 62 97 65 75 53 45 26 57 77 94 14 13 74 52 96 41 36 41 35 66 32 87 64 21 11 44 76 36 44 15 19 34 85
[41] 82  6  6 18 22 78 40 37 40 38 27 23 20 32 18 88 80 12 48 77

The easiest way to calculate the GK gamma statistic is to run the command rcorr.cens(x, y, outx = TRUE), where x is the first vector of numbers you want to compare and y is the second vector of numbers you want to compare. The outx = TRUE command is crucial to the gamma calculation because rcorr.cens does not automatically throw out ties, which are not part of the gamma calculation. Specifying outx = TRUE will throw out ties and give you a true gamma statistic.

Here is an implementation of the code using the fok and recog data we generated a second ago:

# Generate gamma correlation
rcorr.cens(recog, fok, outx = TRUE)
       C Index            Dxy           S.D.              n        missing     uncensored Relevant Pairs     Concordant 
     0.4420697     -0.1158605      0.1552779     60.0000000      0.0000000     60.0000000   1778.0000000    786.0000000 
     Uncertain 
     0.0000000 

Notice that the output gives you explicit data about the calculation being made, including pairs that were thrown out due to “ties”, the number of concordant pairs, and so on. The most imporant piece of output here is Dxy, which is the gamma statistic. For this example, the generated data have a weak negative relationship (-0.116), but if you do this on your own, you will generate a different statistic.2

The output is useful, but for batch commands, sorting through the numbers and copying + pasting can be tedious. A better way to compute the gamma statistic is to specify that you only want the second piece of information generated (i.e., Dxy):

# Generate gamma statistic only
gamma = rcorr.cens(recog, fok, outx = TRUE)[2]
# Display gamma statistic
print(paste0("The gamma correlation is: ", gamma))
[1] "The gamma correlation is: -0.115860517435321"

By specifying the information from the computation that you want to save (i.e., the gamma statistic, or [2]), you are telling R to save it as a scalar, making it easier to write to file.

Alternatives

While using Hmisc seems to be the easiest way to calculate and store gamma correlations for large datasets, there are other ways to calculate gammas in R. The vcdExtra package has a function called GKgamma that performs the same operation:

# Initialize vcdExtra, or install if needed
# install.packages(vcdExtra)
require(vcdExtra)
# Make FOK data into a table
gammaTable <- table(recog,fok)
# Compute gamma using GKgamma
GKgamma(gammaTable)
gamma        : -0.116 
std. error   : 0.155 
CI           : -0.42 0.188 

GKgamma requires that the dataset be constructed as a 2-column table in order to run the calculation, which is why recog and fok are collapsed into gammaTable in the code. The package also does not give you all of the specific information that rcorr.cens does (as demonstrated in the output), but you can save the statistic as a scalar to write to file:

# Generate gamma statistic only
gamma2 = GKgamma(gammaTable)[1]
# Display gamma statistic
print(paste0("The gamma correlation is: ", gamma2))
[1] "The gamma correlation is: -0.115860517435321"


Example With Real Data

Below is an example with real data collected from an experiment conducted at the Georgia Institute of Technology. The dataset, which I have named subjdata, is stored in a CSV file, which I loaded outside of the code presented above. Here is the data from that participant:

print(subjdata)

Ignore the “Recall”, “Recog”, and “Err” columns for now. We will just be calculating the gamma correlation between recognition memory accuracy (“Recog”) and confidence judgments (“CJ”) for the first example.

The calcuation here is the same as before, but with slighly different notation: Because we are pulling data from a table instead of vectors, we have to specify the columns from subjdata that we want to analyze. For CJs, we will put in subjdata$CJ, and for the recognition data, we will put in subjdata$Recog:

# Compute gamma statistic
rcorr.cens(subjdata$Recog, subjdata$CJ, outx = TRUE)
       C Index            Dxy           S.D.              n        missing     uncensored Relevant Pairs     Concordant 
     0.8259494      0.6518987      0.1333313     40.0000000      0.0000000     40.0000000    632.0000000    522.0000000 
     Uncertain 
     0.0000000 

As you can see, this particular participant’s CJs were well-aligned with actual recognition memory performance, yielding a moderately high positive gamma correlation (0.652).

More can be done with the data, such as computing the correlation between FOKs and recognition memory performance for unrecalled items. To do this, I must select the data associated with unrecalled items as a subset:

# Select FOKs for unrecalled items
x <- as.numeric(unlist(subset(subjdata, Recall == 0, select=c(FOK))))
# Select recognition data for unrecalled items
y <- as.numeric(unlist(subset(subjdata, Recall == 0, select=c(Recog))))
# Compute gamma correlation
rcorr.cens(x, y, outx = TRUE)
       C Index            Dxy           S.D.              n        missing     uncensored Relevant Pairs     Concordant 
     0.6034483      0.2068966      0.2521036     22.0000000      1.0000000     22.0000000    232.0000000    140.0000000 
     Uncertain 
     0.0000000 

The correlation here is less impressive (0.207), but does show that this particular participant has some understanding of their own future recognition memory performance when giving an FOK for an unrecalled item, as indicated by the weak positive relationship.


  1. JOLs are prospective memory judgments made during study to gage confidence in future recall performance and are compared to recall outcomes. FOKs are also prospective memory judgments, but are made about future recognition performance during recall. CJs are retrospective memory judgments and gage confidence that a recalled or recognized item is correct.

  2. Note that because these are randomly generated data, the FOK and recognition data will essentially have no relationship to each other, i.e. \(G = 0\).

LS0tDQp0aXRsZTogIkdhbW1hIENvcnJlbGF0aW9uIENhbGN1bGF0aW9ucyBpbiBSIg0Kb3V0cHV0OiANCiAgaHRtbF9ub3RlYm9vazogDQogICAgaGlnaGxpZ2h0OiBweWdtZW50cw0KICAgIHRoZW1lOiBzcGFjZWxhYg0KICAgIHRvYzogeWVzDQotLS0NCioqKg0KDQpUaGlzIGRvY3VtZW50IGlzIGEgdHV0b3JpYWwgb24gaG93IHRvIGNhbGN1bGF0ZSBHb29kbWFuLUtydXNrYWwgZ2FtbWEgY29ycmVsYXRpb25zIGluIFIgYW5kIFJTdHVkaW8uIFRoZXNlIGNhbGN1bGF0aW9ucyBhcmUgaW1wb3J0YW50IGZvciBtZXRhY29nbml0aXZlIHJlc2VhcmNoIGluIHdoaWNoIGp1ZGdtZW50cyBvZiBsZWFybmluZyAoSk9McyksIGZlZWxpbmdzIG9mIGtub3dpbmcgKEZPS3MpLCBhbmQgY29uZmlkZW5jZSBqdWRnbWVudHMgKENKcykgYXJlIGNvbXBhcmVkIHRvIG1lbW9yeSBwZXJmb3JtYW5jZSB0byBhc3Nlc3MgcmVsYXRpdmUgYWNjdXJhY3kuXltKT0xzIGFyZSAqcHJvc3BlY3RpdmUgbWVtb3J5IGp1ZGdtZW50cyogbWFkZSBkdXJpbmcgc3R1ZHkgdG8gZ2FnZSBjb25maWRlbmNlIGluIGZ1dHVyZSByZWNhbGwgcGVyZm9ybWFuY2UgYW5kIGFyZSBjb21wYXJlZCB0byByZWNhbGwgb3V0Y29tZXMuIEZPS3MgYXJlIGFsc28gcHJvc3BlY3RpdmUgbWVtb3J5IGp1ZGdtZW50cywgYnV0IGFyZSBtYWRlIGFib3V0ICpmdXR1cmUgcmVjb2duaXRpb24gcGVyZm9ybWFuY2UqIGR1cmluZyByZWNhbGwuIENKcyBhcmUgKnJldHJvc3BlY3RpdmUgbWVtb3J5IGp1ZGdtZW50cyogYW5kIGdhZ2UgY29uZmlkZW5jZSB0aGF0IGEgcmVjYWxsZWQgb3IgcmVjb2duaXplZCBpdGVtIGlzIGNvcnJlY3QuXSAgDQoNCg0KIyMjIFdoYXQgYXJlIEdhbW1hIENvcnJlbGF0aW9ucz8NCg0KU2ltcGx5IHB1dCwgZ2FtbWEgY29ycmVsYXRpb25zIGFyZSBhIG1lYXN1cmUgb2Ygb3JkaW5hbCBhc3NvY2lhdGlvbiBiZXR3ZWVuIHR3byB2ZWN0b3JzIG9mIG51bWJlcnMuIEluc3RlYWQgb2YgbWVhc3VyaW5nIHRoZSByZWxhdGlvbnNoaXAgYmV0d2VlbiB0d28gc2V0cyBvZiBjb250aW51b3VzIG9yIGludGVydmFsIGRhdGEsIHN1Y2ggYXMgd2hhdCBhICJub3JtYWwiIFBlYXJzb24gY29ycmVsYXRpb24gZG9lcywgdGhlIGdhbW1hIGNvcnJlbGF0aW9uIHRyYW5zbGF0ZXMgc2NhbGUgbnVtYmVycyBpbnRvICBvcmRpbmFsICgicmFuayIpIGRhdGEuIFRodXMsIHNjYWxlIG1ldGFjb2duaXRpdmUganVkZ21lbnRzLCBzdWNoIGFzIEpPTHMgcmFuZ2luZyBmcm9tIDAgLSAxMDAsIGFyZSBub3QgYW5hbHl6ZWQgYnkgdGhlaXIgYWJzb2x1dGUgdmFsdWVzOyByYXRoZXIsIHRoZXkgYXJlIGNvbnNpZGVyZWQgcmFuayBqdWRnbWVudHMgb2YgbWVtb3J5IGxpa2VsaWhvb2QuIEEganVkZ21lbnQgb2YgIjEwIiwgZm9yIGV4YW1wbGUsIHdvdWxkIGJlIGNvbnNpZGVyZWQgYSBncmVhdGVyIHBlcmNpZXZlZCBsaWtlbGlob29kIHRoYW4gYSBqdWRnbWVudCBvZiAiMSIsIGJ1dCB3b3VsZCBub3QgYmUgY29uc2lkZXJlZCAxMCB0aW1lcyBtb3JlIGxpa2VseSB0aGFuICIxIiBpbiBhIGdhbW1hIGNvcnJlbGF0aW9uLiANCg0KR2FtbWEgY29ycmVsYXRpb25zIGFyZSBjb21wdXRlZCBwYWlyLXdpc2UsIG1lYW5pbmcgdGhhdCB0aGUgbWVtb3J5IG91dGNvbWVzICgwIG9yIDEpIGFyZSBwYWlyZWQgd2l0aCBhIGNvcnJlc3BvbmRpbmcganVkZ21lbnQgKEpPTCwgRk9LLCBldGMuKSBhbmQgY29tcGFyZWQgdG8gYWxsIG90aGVyIHBhaXJzIG9mIG51bWJlcnMuIEFsbCBwYWlycyB0aGF0IGFyZSByYW5rZWQgb24gdGhlIHNhbWUgb3JkZXIgZm9yIGJvdGggdmFyaWFibGVzIGFyZSBjb25zaWRlcmVkICJjb25jb3JkYW50IiwgYW5kIGFsbCBwYWlycyB0aGF0IGFyZSByYW5rZWQgb24gdGhlIHJldmVyc2Ugb3JkZXIgYXJlIGNvbnNpZGVyZWQgImRpc2NvbmNvcmRhbnQiLiBJbnN0YW5jZXMgYXJlIHRhbGxpZWQsIGFuZCB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBjb25jb3JkYW50IGFuZCBkaXNjb25jb3JkYW50IHBhaXJzIGlzIGRpdmlkZWQgYnkgdGhlIHN1bSBvZiB0aGUgY29uY29yZGFudCBhbmQgZGlzY29uY29yZGFudCBwYWlycyB0byByZXR1cm4gYSB2ZWN0b3IgZnJvbSAtMSB0byAxLCB3aXRoIHZhbHVlcyBsZXNzIHRoYW4gMCBpbmRpY2F0aW5nIGEgbmVnYXRpdmUgcmVsYXRpb25zaGlwIGJldHdlZW4gdGhlIGp1ZGdtZW50IGFuZCBzdWJzZXF1ZW50IG1lbW9yeSAoZS5nLiBsYXJnZXIgSk9McyBmb3IgaXRlbXMgdGhhdCBhcmUgdW5yZWNhbGxlZCkgYW5kIHZhbHVlcyBncmVhdGVyIHRoYW4gMCBpbmRpY2F0aW5nIGEgcG9zaXRpdmUgcmVsYXRpb25zaGlwIChlLmcuIGxhcmdlciBKT0xzIGZvciBpdGVtcyB0aGF0IGFyZSByZWNhbGxlZCkuDQoNCiRHID0gXGZyYWN7Tl9jIC0gTl9kfXtOX2MgKyBOX2R9JCAgDQoNCg0KIyMjIENhbGN1bGF0aW9ucyBpbiBSDQoNCipBcyBhIG5vdGUqOiBJZiB5b3UgYXJlIHZpZXdpbmcgdGhpcyBpbiBSIGFzIGEgTm90ZWJvb2ssIHRoZW4geW91IGNhbiBleGVjdXRlIHRoZSBjb2RlIGluIHRoZSBib3hlcyBieSBjbGlja2luZyAqUnVuKiBvciBieSBjbGlja2luZyBpbnNpZGUgb2YgdGhlIGNodW5rIGFuZCBwcmVzc2luZyAqQ3RybCtTaGlmdCtFbnRlciouIElmIHlvdSBhcmUgdmlld2luZyB0aGlzIGFzIGEgUERGIG9yIEhUTUwsIHRoZSBvdXRwdXQgd2lsbCBhcHBlYXIgYmVsb3cgdGhlIGFzc29jaWF0ZWQgYm94Lg0KDQpUaGUgZmlyc3QgdGhpbmcgdGhhdCB5b3Ugd2lsbCBuZWVkIHRvIGRvIChhc3N1bWluZyB0aGF0IHlvdSBhbHJlYWR5IGhhdmUgUlN0dWRpbyBzZXQgdXApIGlzIGluc3RhbGwgYW5kIGluaXRpYWxpemUgdGhlIHBhY2thZ2VzIGFzc29jaWF0ZWQgd2l0aCBnYW1tYXMgYW5kIHRoZSBjb21wdXRhdGlvbnMgYmVsb3cuIFRoZSBjb2RlIGhlcmUgd2lsbCBvbmx5IGluaXRpYWxpemUgaXQsIGJ1dCBpbnN0YWxsYXRpb24gY29kZSBpcyBhdmFpbGFibGUgaW4gdGhlIGNvbW1lbnRlZCBzZWN0aW9ucy4NCg0KYGBge3J9DQojaW5zdGFsbC5wYWNrYWdlcyhIbWlzYykNCnJlcXVpcmUoSG1pc2MpDQpgYGANCg0KKkhtaXNjKiBpcyBhIGZyZWVseS1hdmFpbGFibGUgcGFja2FnZSAoYWJicmV2aWF0ZWQgZnJvbSAiW0hhcnJlbGwgTWlzY2VsbGFuZW91c10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL0htaXNjL2luZGV4Lmh0bWwpIikgdGhhdCBzdXBwbGVtZW50cyB0aGUgc3RhbmRhcmQgUiBzdGF0aXN0aWNzIHJlcG9zaXRvcnkuIE1vc3Qgb2YgdGhlIGZ1bmN0aW9ucyB0aGF0IGFyZSBhdmFpbGFibGUgaGVyZSBhcmUgb2Ygbm8gaW50ZXJlc3QgdG8gdXMsIGV4Y2VwdCBmb3IgYHJjb3JyLmNlbnNgLCB3aGljaCBwcm92aWRlcyB0aGUgR29vZG1hbi1LcnVza2FsIGNvcnJlbGF0aW9uIC0tIHByb3ZpZGVkIHRoYXQgeW91IHVzZSB0aGUgY29ycmVjdCBjb2RlLiBXZSB3aWxsIGdldCBpbnRvIHRoYXQgbGF0ZXIuDQoNClRvIGJlZ2luLCBsZXQncyBnZW5lcmF0ZSBhIGRhdGFzZXQgdG8gd29yayB3aXRoLiBGb3IgdGhpcyBleGFtcGxlLCB3ZSB3aWxsIGJlIHVzaW5nIHNjYWxlIEZPS3MgZnJvbSAwIHRvIDEwMCwgd2l0aCBhbiBGT0sgb2YgMCBtZWFuaW5nIG5vIGNvbmZpZGVuY2UgYW5kIGFuIEZPSyBvZiAxMDAgbWVhbmluZyBhYnNvbHV0ZSBjb25maWRlbmNlLCBhbmQgcmVjb2duaXRpb24gb3V0Y29tZSBkYXRhLCB3aXRoIDAgbWVhbmluZyBhbiBpbmNvcnJlY3QgcmVjb2duaXRpb24gYW5kIDEgbWVhbmluZyBhIGNvcnJlY3QgcmVjb2duaXRpb24uIFRoZSBjb2RlIGJlbG93IHJhbmRvbWx5IGdlbmVyYXRlcyA2MCBGT0tzIGZyb20gMCAtIDEwMCBhbmQgNjAgcmVjb2duaXRpb24gbWVtb3J5IG91dGNvbWVzOg0KDQoNCmBgYHtyfQ0KIyBSYW5kb21seSBnZW5lcmF0ZSBGT2sgZGF0YXNldA0KDQpyZWNvZyA8LSBzYW1wbGUoMDoxLDYwLHJlcGxhY2U9VFJVRSkNCmZvayA8LSBzYW1wbGUoMDoxMDAsNjAscmVwbGFjZT1UUlVFKQ0KDQojIFNob3cgZ2VuZXJhdGVkIGRhdGENCnByaW50KHJlY29nKQ0KcHJpbnQoZm9rKQ0KYGBgDQogICANCiAgIA0KVGhlIGVhc2llc3Qgd2F5IHRvIGNhbGN1bGF0ZSB0aGUgR0sgZ2FtbWEgc3RhdGlzdGljIGlzIHRvIHJ1biB0aGUgY29tbWFuZCBgcmNvcnIuY2Vucyh4LCB5LCBvdXR4ID0gVFJVRSlgLCB3aGVyZSBgeGAgaXMgdGhlIGZpcnN0IHZlY3RvciBvZiBudW1iZXJzIHlvdSB3YW50IHRvIGNvbXBhcmUgYW5kIGB5YCBpcyB0aGUgc2Vjb25kIHZlY3RvciBvZiBudW1iZXJzIHlvdSB3YW50IHRvIGNvbXBhcmUuIFRoZSBgb3V0eCA9IFRSVUVgIGNvbW1hbmQgaXMgY3J1Y2lhbCB0byB0aGUgZ2FtbWEgY2FsY3VsYXRpb24gYmVjYXVzZSBgcmNvcnIuY2Vuc2AgZG9lcyBub3QgYXV0b21hdGljYWxseSB0aHJvdyBvdXQgdGllcywgd2hpY2ggYXJlIG5vdCBwYXJ0IG9mIHRoZSBnYW1tYSBjYWxjdWxhdGlvbi4gU3BlY2lmeWluZyBgb3V0eCA9IFRSVUVgIHdpbGwgdGhyb3cgb3V0IHRpZXMgYW5kIGdpdmUgeW91IGEgdHJ1ZSBnYW1tYSBzdGF0aXN0aWMuDQoNCkhlcmUgaXMgYW4gaW1wbGVtZW50YXRpb24gb2YgdGhlIGNvZGUgdXNpbmcgdGhlIGBmb2tgIGFuZCBgcmVjb2dgIGRhdGEgd2UgZ2VuZXJhdGVkIGEgc2Vjb25kIGFnbzoNCg0KYGBge3J9DQojIEdlbmVyYXRlIGdhbW1hIGNvcnJlbGF0aW9uDQoNCnJjb3JyLmNlbnMocmVjb2csIGZvaywgb3V0eCA9IFRSVUUpDQpgYGANCg0KTm90aWNlIHRoYXQgdGhlIG91dHB1dCBnaXZlcyB5b3UgZXhwbGljaXQgZGF0YSBhYm91dCB0aGUgY2FsY3VsYXRpb24gYmVpbmcgbWFkZSwgaW5jbHVkaW5nIHBhaXJzIHRoYXQgd2VyZSB0aHJvd24gb3V0IGR1ZSB0byAidGllcyIsIHRoZSBudW1iZXIgb2YgY29uY29yZGFudCBwYWlycywgYW5kIHNvIG9uLiBUaGUgbW9zdCBpbXBvcmFudCBwaWVjZSBvZiBvdXRwdXQgaGVyZSBpcyBgRHh5YCwgd2hpY2ggaXMgdGhlIGdhbW1hIHN0YXRpc3RpYy4gRm9yIHRoaXMgZXhhbXBsZSwgdGhlIGdlbmVyYXRlZCBkYXRhIGhhdmUgYSB3ZWFrIG5lZ2F0aXZlIHJlbGF0aW9uc2hpcCAoLTAuMTE2KSwgYnV0IGlmIHlvdSBkbyB0aGlzIG9uIHlvdXIgb3duLCB5b3Ugd2lsbCBnZW5lcmF0ZSBhIGRpZmZlcmVudCBzdGF0aXN0aWMuXltOb3RlIHRoYXQgYmVjYXVzZSB0aGVzZSBhcmUgcmFuZG9tbHkgZ2VuZXJhdGVkIGRhdGEsIHRoZSBGT0sgYW5kIHJlY29nbml0aW9uIGRhdGEgd2lsbCBlc3NlbnRpYWxseSBoYXZlIG5vIHJlbGF0aW9uc2hpcCB0byBlYWNoIG90aGVyLCBpLmUuICRHID0gMCQuXQ0KDQpUaGUgb3V0cHV0IGlzIHVzZWZ1bCwgYnV0IGZvciBiYXRjaCBjb21tYW5kcywgc29ydGluZyB0aHJvdWdoIHRoZSBudW1iZXJzIGFuZCBjb3B5aW5nICsgcGFzdGluZyBjYW4gYmUgdGVkaW91cy4gQSBiZXR0ZXIgd2F5IHRvIGNvbXB1dGUgdGhlIGdhbW1hIHN0YXRpc3RpYyBpcyB0byBzcGVjaWZ5IHRoYXQgeW91IG9ubHkgd2FudCB0aGUgc2Vjb25kIHBpZWNlIG9mIGluZm9ybWF0aW9uIGdlbmVyYXRlZCAoaS5lLiwgYER4eWApOg0KDQpgYGB7cn0NCiMgR2VuZXJhdGUgZ2FtbWEgc3RhdGlzdGljIG9ubHkNCg0KZ2FtbWEgPSByY29yci5jZW5zKHJlY29nLCBmb2ssIG91dHggPSBUUlVFKVsyXQ0KDQoNCiMgRGlzcGxheSBnYW1tYSBzdGF0aXN0aWMNCg0KcHJpbnQocGFzdGUwKCJUaGUgZ2FtbWEgY29ycmVsYXRpb24gaXM6ICIsIGdhbW1hKSkNCmBgYA0KDQpCeSBzcGVjaWZ5aW5nIHRoZSBpbmZvcm1hdGlvbiBmcm9tIHRoZSBjb21wdXRhdGlvbiB0aGF0IHlvdSB3YW50IHRvIHNhdmUgKGkuZS4sIHRoZSBnYW1tYSBzdGF0aXN0aWMsIG9yIGBbMl1gKSwgeW91IGFyZSB0ZWxsaW5nIFIgdG8gc2F2ZSBpdCBhcyBhIHNjYWxhciwgbWFraW5nIGl0IGVhc2llciB0byB3cml0ZSB0byBmaWxlLg0KDQoNCiMjIyBBbHRlcm5hdGl2ZXMNCg0KV2hpbGUgdXNpbmcgKkhtaXNjKiBzZWVtcyB0byBiZSB0aGUgZWFzaWVzdCB3YXkgdG8gY2FsY3VsYXRlIGFuZCBzdG9yZSBnYW1tYSBjb3JyZWxhdGlvbnMgZm9yIGxhcmdlIGRhdGFzZXRzLCB0aGVyZSBhcmUgb3RoZXIgd2F5cyB0byBjYWxjdWxhdGUgZ2FtbWFzIGluIFIuIFRoZSAqdmNkRXh0cmEqIFtwYWNrYWdlXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvdmNkRXh0cmEvaW5kZXguaHRtbCkgaGFzIGEgZnVuY3Rpb24gY2FsbGVkIGBHS2dhbW1hYCB0aGF0IHBlcmZvcm1zIHRoZSBzYW1lIG9wZXJhdGlvbjoNCg0KYGBge3J9DQojIEluaXRpYWxpemUgdmNkRXh0cmEsIG9yIGluc3RhbGwgaWYgbmVlZGVkDQojIGluc3RhbGwucGFja2FnZXModmNkRXh0cmEpDQpyZXF1aXJlKHZjZEV4dHJhKQ0KDQojIE1ha2UgRk9LIGRhdGEgaW50byBhIHRhYmxlDQpnYW1tYVRhYmxlIDwtIHRhYmxlKHJlY29nLGZvaykNCg0KIyBDb21wdXRlIGdhbW1hIHVzaW5nIEdLZ2FtbWENCkdLZ2FtbWEoZ2FtbWFUYWJsZSkNCg0KYGBgDQoNCmBHS2dhbW1hYCByZXF1aXJlcyB0aGF0IHRoZSBkYXRhc2V0IGJlIGNvbnN0cnVjdGVkIGFzIGEgMi1jb2x1bW4gdGFibGUgaW4gb3JkZXIgdG8gcnVuIHRoZSBjYWxjdWxhdGlvbiwgd2hpY2ggaXMgd2h5IGByZWNvZ2AgYW5kIGBmb2tgIGFyZSBjb2xsYXBzZWQgaW50byBgZ2FtbWFUYWJsZWAgaW4gdGhlIGNvZGUuIFRoZSBwYWNrYWdlIGFsc28gZG9lcyBub3QgZ2l2ZSB5b3UgYWxsIG9mIHRoZSBzcGVjaWZpYyBpbmZvcm1hdGlvbiB0aGF0IGByY29yci5jZW5zYCBkb2VzIChhcyBkZW1vbnN0cmF0ZWQgaW4gdGhlIG91dHB1dCksIGJ1dCB5b3UgY2FuIHNhdmUgdGhlIHN0YXRpc3RpYyBhcyBhIHNjYWxhciB0byB3cml0ZSB0byBmaWxlOg0KDQpgYGB7cn0NCiMgR2VuZXJhdGUgZ2FtbWEgc3RhdGlzdGljIG9ubHkNCmdhbW1hMiA9IEdLZ2FtbWEoZ2FtbWFUYWJsZSkNCiMgRGlzcGxheSBnYW1tYSBzdGF0aXN0aWMNCnByaW50KHBhc3RlMCgiVGhlIGdhbW1hIGNvcnJlbGF0aW9uIGlzOiAiLCBnYW1tYTIpKQ0KYGBgDQoNCioqKg0KKioqDQoNCiMjIEV4YW1wbGUgV2l0aCBSZWFsIERhdGENCg0KQmVsb3cgaXMgYW4gZXhhbXBsZSB3aXRoIHJlYWwgZGF0YSBjb2xsZWN0ZWQgZnJvbSBhbiBleHBlcmltZW50IGNvbmR1Y3RlZCBhdCB0aGUgR2VvcmdpYSBJbnN0aXR1dGUgb2YgVGVjaG5vbG9neS4gVGhlIGRhdGFzZXQsIHdoaWNoIEkgaGF2ZSBuYW1lZCBgc3ViamRhdGFgLCBpcyBzdG9yZWQgaW4gYSBDU1YgZmlsZSwgd2hpY2ggSSBsb2FkZWQgb3V0c2lkZSBvZiB0aGUgY29kZSBwcmVzZW50ZWQgYWJvdmUuIEhlcmUgaXMgdGhlIGRhdGEgZnJvbSB0aGF0IHBhcnRpY2lwYW50Og0KDQpgYGB7cn0NCnByaW50KHN1YmpkYXRhKQ0KYGBgDQpJZ25vcmUgdGhlICJSZWNhbGwiLCAiUmVjb2ciLCBhbmQgIkVyciIgY29sdW1ucyBmb3Igbm93LiBXZSB3aWxsIGp1c3QgYmUgY2FsY3VsYXRpbmcgdGhlIGdhbW1hIGNvcnJlbGF0aW9uIGJldHdlZW4gcmVjb2duaXRpb24gbWVtb3J5IGFjY3VyYWN5ICgiUmVjb2ciKSBhbmQgY29uZmlkZW5jZSBqdWRnbWVudHMgKCJDSiIpIGZvciB0aGUgZmlyc3QgZXhhbXBsZS4NCg0KVGhlIGNhbGN1YXRpb24gaGVyZSBpcyB0aGUgc2FtZSBhcyBiZWZvcmUsIGJ1dCB3aXRoIHNsaWdobHkgZGlmZmVyZW50IG5vdGF0aW9uOiBCZWNhdXNlIHdlIGFyZSBwdWxsaW5nIGRhdGEgZnJvbSBhIHRhYmxlIGluc3RlYWQgb2YgdmVjdG9ycywgd2UgaGF2ZSB0byBzcGVjaWZ5IHRoZSBjb2x1bW5zIGZyb20gYHN1YmpkYXRhYCB0aGF0IHdlIHdhbnQgdG8gYW5hbHl6ZS4gRm9yIENKcywgd2Ugd2lsbCBwdXQgaW4gYHN1YmpkYXRhJENKYCwgYW5kIGZvciB0aGUgcmVjb2duaXRpb24gZGF0YSwgd2Ugd2lsbCBwdXQgaW4gYHN1YmpkYXRhJFJlY29nYDoNCg0KYGBge3J9DQojIENvbXB1dGUgZ2FtbWEgc3RhdGlzdGljDQpyY29yci5jZW5zKHN1YmpkYXRhJFJlY29nLCBzdWJqZGF0YSRDSiwgb3V0eCA9IFRSVUUpDQpgYGANCg0KQXMgeW91IGNhbiBzZWUsIHRoaXMgcGFydGljdWxhciBwYXJ0aWNpcGFudCdzIENKcyB3ZXJlIHdlbGwtYWxpZ25lZCB3aXRoIGFjdHVhbCByZWNvZ25pdGlvbiBtZW1vcnkgcGVyZm9ybWFuY2UsIHlpZWxkaW5nIGEgbW9kZXJhdGVseSBoaWdoIHBvc2l0aXZlIGdhbW1hIGNvcnJlbGF0aW9uICgwLjY1MikuDQoNCk1vcmUgY2FuIGJlIGRvbmUgd2l0aCB0aGUgZGF0YSwgc3VjaCBhcyBjb21wdXRpbmcgdGhlIGNvcnJlbGF0aW9uIGJldHdlZW4gRk9LcyBhbmQgcmVjb2duaXRpb24gbWVtb3J5IHBlcmZvcm1hbmNlIGZvciAqdW5yZWNhbGxlZCogaXRlbXMuIFRvIGRvIHRoaXMsIEkgbXVzdCBzZWxlY3QgdGhlIGRhdGEgYXNzb2NpYXRlZCB3aXRoIHVucmVjYWxsZWQgaXRlbXMgYXMgYSBzdWJzZXQ6DQoNCmBgYHtyfQ0KIyBTZWxlY3QgRk9LcyBmb3IgdW5yZWNhbGxlZCBpdGVtcw0KeCA8LSBhcy5udW1lcmljKHVubGlzdChzdWJzZXQoc3ViamRhdGEsIFJlY2FsbCA9PSAwLCBzZWxlY3Q9YyhGT0spKSkpDQojIFNlbGVjdCByZWNvZ25pdGlvbiBkYXRhIGZvciB1bnJlY2FsbGVkIGl0ZW1zDQp5IDwtIGFzLm51bWVyaWModW5saXN0KHN1YnNldChzdWJqZGF0YSwgUmVjYWxsID09IDAsIHNlbGVjdD1jKFJlY29nKSkpKQ0KIyBDb21wdXRlIGdhbW1hIGNvcnJlbGF0aW9uDQpyY29yci5jZW5zKHgsIHksIG91dHggPSBUUlVFKQ0KDQpgYGANCg0KVGhlIGNvcnJlbGF0aW9uIGhlcmUgaXMgbGVzcyBpbXByZXNzaXZlICgwLjIwNyksIGJ1dCBkb2VzIHNob3cgdGhhdCB0aGlzIHBhcnRpY3VsYXIgcGFydGljaXBhbnQgaGFzIHNvbWUgdW5kZXJzdGFuZGluZyBvZiB0aGVpciBvd24gZnV0dXJlIHJlY29nbml0aW9uIG1lbW9yeSBwZXJmb3JtYW5jZSB3aGVuIGdpdmluZyBhbiBGT0sgZm9yIGFuIHVucmVjYWxsZWQgaXRlbSwgYXMgaW5kaWNhdGVkIGJ5IHRoZSB3ZWFrIHBvc2l0aXZlIHJlbGF0aW9uc2hpcC4NCg==