Comparative analysis of ChatGPT and clinical visits

Conceptual questions

1. Why so few observations and how were they sampled?

2. How are observations adjudicated to the “correct” diagnosis? Are we assuming that the diagnoses from the visit are 100% accurate?

3. How variable are the outcomes between ophtalmologists? Given a health record, typical to this population, how likely it is that different outcomes will be observed from two ophtalmologists?

Proposal questions

1. In Data Collection you write “Data will be extracted from electronic medical records”.

1. by who and how?
2. how accurately?

2. Are the background data standardized in some way?

3. How are you planning to “evaluate accuracy” without knowing the true condition?

Data questions

1. You wrote in WhatsApp that the data will be recoded into letters. What does it mean?

2. The example data contain diagnoses/suggestions from only one entity, and it is unclear whether this is a physician or chatGPT.

Modeling remarks

1. Logistic regression will be cumbersome or even impossible to fit (depends on the data). Unless you have a specific model in mind, I suggest other, simpler tests for comparison.

2. I believe that your actual objective is to test if chatGPT is non-inferior to the clinical visits. Testing that is more feasible with n = 100.