Introduction

The Perl 6 user survey took place in the last days of June 2018. 220 developers provided answers in a variety of topics, that went from their relationship with the language to other aspects, among which the most important for me, and the focus of this report, is the documentation score.

The Perl 6 documentation, as the rest of the Perl 6 project, is an almost totally volunteer effort whose development takes place in a GitHub repository. Perception of quality is important, so that is why we will try to analyze, in this report, how the documentation is scored among the survey respondents. But first, the summary:

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    1.00    6.00    7.00    6.87    8.00   10.00       5

So the average is 6.8697674, with the median on 7. It’s a low grade, and we are interested in finding some causes so that it can be improved in the future.

Evolution of score with responses

The diffusion of the survey was first made among the participants in the #perl6 IRC channel, and then through different media: Twitter, Facebook, eventually making it to the Perl6 Weekly and mailing list. We will try to consider time of response a variable, and analyze how score changes with it. For this response, score was an integer between 0 and 10.

This moving average was computed by averaging 30 consecutive scores, and it clearly shows there’s a certain dynamic to it. The score goes initially up, then slightly down, then it goes on a deep dive to retrieve a high again, to eventually settle around 6.5 in the late comers. Peaks happen around the 50th response, approximately at the time it was spread over Twitter, and again at the 120th response, approximately at the time it showed up on the Perl 6 weekly. However, there are deep dives here and there showing an average of less than 6.5; the scores given by the inner circle, those that responded the first, are consistently below 7. This might indicate that people who are familiar with the language get less mileage out of the documentation than newcomers. But we will have to cross-check this looking at other variables.

Cross-checking against community score

Perl 6 values highly its community, which in fact is the source of the language. Let us cross-check both scores.

This chart roughly plots community score (in the y axis) against documentation score (in the x axis). Most responses concentrate on the high-community score, high documentation score. However, the later is consistently lower than the former, although there are a few points that could safely be removed and not considered; it might even be the case that it is a repeated answer by the same person. Let us look at a boxplot of the same quantities, but swapping the axes

In this case, the documentation score roughly increases with the community score, but averages are around 2 points below. But this allows us to filter out people that are kind of unhappy with the whole thing. For instance, those giving a 6 to the community give the same score to the documentation, same with 7, although there is a gap there with not many respondents.

General idea, however, is that score is valued lower than the community.

Documentation score against experience

This is interesting in the sense that people who give it the lowest score are those that are not using Perl 6 anymore. Correlation is not causation, but somehow that response is less interesting than the rest of them. Median value for people that have just started or have been there for less than a month is 7, which is not excellent, but neither too bad. People who have been less than a year value it at 8, but there are a few outliers. These scores would reflect the evolution of the documentation quality with time. Those with more experience giving it a score lower than newcomers would indicate some improvement in the documentation. There’s some hint that that would be happening: people with less than a month experience don’t give it low scores; however, some that have just started give it very low score. The signal is not consistent, although it is the fact that we need to improve the documentation.

Conclusions

Even filtering out possible trolls and taking into account experience with the language, the score achieved by the documentation hovers around 7, which is still a low score.

This report is, however, a snapshot of the situation this year that will be useful once this survey is repeated periodically. At least we have a data point now that we can use to compare and evaluate changes in the documentation. And, for that matter, in the rest of the Perl 6 world.