What is the role of inheritance in the problem solving abilities of individuals?
Pierce Edmiston
What individuals can do and learn to do today critically depends on what others were able to do yesterday. Humans are smart, but we are also hoarders; human cultures accumulate artifacts, ideas, and information far more than any other animal. Part of what makes this accumulation helpful is that people never have to start from scratch when solving a problem. Inventors, for example, inherit and adapt existing solutions to problems so pervasively as to make the history of technology one of near continuous change, with each new invention depending heavily on its precursors (Basalla, 1989). But what this historical evidence does not provide is a way to understand more directly the role inheritance may play in the problem solving abilities of individuals. Does inheriting someone else’s solution just give a head start on a problem or does it actually make for more effective problem solving? The purpose of this research is to better understand the role of inheritance in the problem solving ability of individuals.
To answer this question, I consider the simplest form of inheritance where one person works on a solution to a problem and passes it on to a second person to serve as the starting point for continued work. Wikipedia article writing is an example of this type of teamwork. My hypothesis is that organizing teamwork around inheritance or diachronic teamwork will result in more effective problem solving than other strategies for allocating the same number of labor hours (Experiment 1). The two alternative strategies that are considered in Experiment 1 are (a) individuals working alone for the same total amount of time (solo “teams”) and (b) teams of two working together at the same time (synchronic versus diachronic). The remaining experiments are designed to test the reasons and conditions under which inheritance leads to more effective problem solving. The first proposed reason is that organizing teamwork around inheritance makes solutions more responsive to feedback (Experiments 2 and 3). When solutions are inherited rather than designed, improvements are likely to be incremental and thus dependent on informative sources of feedback. The second proposed reason is that inheritance is most effective when the members of the team have overlapping skills. This question is posed as a modeling question to be reported as Experiment 4. Experiment 5 is an important extension of Experiment 1 that addresses the discrepancy between calendar hours and labor hours in the team structure manipulation by having diachronic and synchronic teams complete multiple problems in the same number of calendar hours. Together these results would suggest that the tools and abilities that facilitate inheritance may have served to bootstrap the problem solving abilities of humans.
Teams are allotted the same number of labor hours to solve a problem. In diachronic teams, the experimental condition, team members work one at a time on a solution, and the second team member inherits the first team member’s solution as a starting point. The two control conditions are solo “teams” where individuals work alone on a solution for the same total amount of time, and synchronic teams who instead of working one at a time, work together at the same time. The extent to which diachronic teams are more effective than either solo or synchronic teams is proposed as a test of the role of inheritance in the problem solving ability of individuals.
Design
| # | Experiment | Team structure | Problem | Feedback |
|---|---|---|---|---|
| 1 | Team structure | diachronic, solo, synchronic | classification | yes |
| 2 | Feedback | diachronic, synchronic | classification | no, enforce |
| 3 | Insight problem | diachronic, synchronic | insight | yes |
| 4 | Overlapping skills | diachronic, synchronic | (modeling) | yes |
| 5 | Multiple problems | diachronic, synchronic | multiple classification | yes |
1. Team structure
Experiment 1 is designed to test the hypothesis that diachronic collaboration is more effective than synchronic collaboration at solving classification problems. The solution to a classification problem involves generating predicted labels or “classes” for unlabeled data using statistics and machine learning to improve the accuracy of the predictions. For example, a typical classification problem is to predict whether or not passengers on the Titanic survived based on their age, gender, ticket price, cabin location, and other features of the passenger. Classification problems are good candidates for comparing the effectiveness of different team structures because there are many possible solutions, all varying in degree of success. For example, teams can use decision trees, linear regression, neural network models, and combinations of these, each varying in which features they are trained on and the parameters of the models. Given the vast space of possible solutions, a useful strategy for solving classification problems is to iteratively try out different solutions and incrementally improve classification accuracy1. Although both diachronic and synchronic teams are able to iteratively develop solutions to classification problems, diachronic teams are hypothesized to be more effective than synchronic teams at utilizing this feedback to iteratively improve classification accuracy.
Procedure. Skilled and motivated participants are randomly assigned team and condition. Each team is given training data and instructed to write a program that accepts unlabeled test data and generates predicted labels. Synchronic teams are given two hours with all four team members in the same room. Each team member is provided a personal work station and network access to the team’s program files, but there are no constraints on how the members of the synchronic teams interact or delegate labor over the course of the two hours. Diachronic teams work one at a time, each spending two hours on the program, and are not allowed to interact with their whole team all at once. Diachronic teamwork is incentivized by explaining diachronic team structures and rewarding participants based on team performance, not on individual performance. For diachronic teams, only a single workstation is needed. In either team, members can at any time submit their program for evaluation against the test data, and receive feedback in the form of a classification accuracy score. After eight labor hours have been spent by each team, the final version of each team’s program is submitted and evaluated. The primary hypothesis tested in Experiment 1 is that diachronic teams will produce more accurate predictions than synchronic teams.
Research questions
Are diachronic teams more effective problem solvers? The primary hypothesis tested in Experiment 1 is that diachronic collaboration will be more effective than synchronic collaborations at solving classification problems. Problem solving effectiveness is measured in terms of the classification accuracy of the team’s final submission. Participants are equally skilled and motivated, in which case a difference in team performance can be attributed to the structure of the team’s interactions.
Do diachronic teams make more submissions? The primary reason diachronic teams are expected to outperform synchronic teams is because they are better able to utilize feedback and iteratively improve solutions. If this is true, we should expect to see diachronic teams make more submissions (and thus receive more feedback) than synchronic teams. Part of the reason diachronic teams may make more submissions reflects the fact that a team’s submission rate may be more dependent on calendar hours than on labor hours. If controlling for submission rate by calendar hours still reveals a difference in submissions for diachronic and synchronic teams, then part of the reason diachronic teams are better able to utilize feedback may be because they able to engage in a faster development cycle.
2. Feedback
In Experiment 2, the feedback made available to teams about their performance is altered in order to test the hypothesis that the effectiveness of diachronic collaboration in solving problems depends on reliable sources of feedback. In the first condition, feedback is removed completely, and teams do not know for certain their performance until the end of the experiment. Removing feedback is expected to impair diachronic more than synchronic team performance. In the second condition, feedback is enforced such that a team members contributions can only be shared with the rest of the team if that contribution measurably improves team performance. If diachronic teams are already using feedback to incrementally improve team performance, enforcing feedback should help synchronic teams more by encouraging them to focus on adaptive and incremental improvements in team performance.
Feedback mechanisms. When feedback is provided (left), teams can submit solutions for evaluation. This is the feedback condition used in Experiment 1. Experiment 2 includes two other feedback conditions. Feedback is either removed (middle) or enforced (right) such that changes to a solution that do not improve solution success are rejected.
Procedure. Teams in Experiment 2 are assigned to either the no feedback condition or the enforced feedback position. In the no feedback condition, diachronic and synchronic teams are no longer able to submit their programs for evaluation prior to the end of the experiment. In the enforced feedback condition, every time a team member submits a change to the team’s program files, the program is evaluated against the test data. If the contribution does not improve past team performance, it is rejected, and returned for further editing. Otherwise, the contribution can be distributed to the other members of the team.
Research questions
Do diachronic teams depend on feedback? If the adaptive problem solving abilities of diachronic teams are due to increased responsiveness to feedback, then removing feedback should impair diachronic team performance relative to synchronic team performance.
Does enforcing feedback improve synchronic performance? If the reason synchronic teams are outperformed by diachronic teams is that they aren’t utilizing feedback as effectively, enforcing feedback so that team performance must improve through all changes to the team’s solution should improve synchronic team performance more than diachronic team performance.
3. Insight problem
Not all problems have informative sources of feedback. Where classification problems have many possible solutions, all varying in degree of success, insight problems have only one solution; they are either solved or not. Experiment 3 investigates the effect of team structure on solving insight problems. Because feedback is unhelpful, diachronic teams do not have an advantage in being able to incrementally improve solutions. For teamwork to be helpful in solving an insight problem, everyone in a team should search for the solution in parallel. This type of problem solving is hypothesized to be better equipped for synchronic collaboration.
The insight problems in Experiment 3 are similarly data-oriented to facilitate comparison across experiments. Participants with the same skills and motivation compete in a “Draw my data” contest. In this contest, teams are given artificial data and asked to identify the object who’s image is contained hidden within the data. Solving this problem requires searching through the many variables of the data (most of them irrelevant) and visualizing various combinations until the correct view of the data is discovered. Discovering the object that was “drawn” in the data is the moment of insight that solves the problem.
Procedure. Teams complete as many “Draw my data” problems as they can in 8 labor hours, organized synchronically or diachronically. Solutions are submitted by naming the object in the data. If correct, teams receive another dataset. The code used to generate the visualizations and the plots themselves are shared among the members of the team.
Research questions
Do synchronic teams excel at insight problems? Without feedback to guide iteration and change, diachronic teams no longer have an advantage over synchronic teams when solving insight problems. Synchronic teams are expected to outperform diachronic teams because they are better able to coordinate search for the solution in parallel.
4. Overlapping skills
One of the proposed reasons for why inheritance might make for more effective problem solving is that inheritance of solutions is an efficient use of labor hours when team members have overlapping skills. However, it is inherently difficult to manipulate the skills of participants coming in to an experiment, and so an alternative approach where problem solving abilities of teams are modeled is used.
Procedure. Individuals are initialized with differing skills in two domains and arranged in either diachronic or synchronic teams to solve some problem. Diachronic teams, since individuals work one at a time, can use both of an individuals skills in order to attempt an improvement of the team’s solution. Synchronic teams, since individuals have to work at the same time, delegate problem solving by domain to the most qualified team member. I intend to show that as the skill level of the team members overlaps, diachronic collaboration becomes more effective than synchronic collaboration. Conversely, when team members differ widely in skills, synchronic collaboration is expected to become more effective.
5. Multiple problems
The purpose of Experiment 4 is to replicate Experiment 1 and extend it to include teams solving multiple adaptive problems over the same number of calendar hours. In Experiment 1, synchronic teams completed their task in 2 calendar hours, where diachronic teams required 8. Even if diachronic teams outperformed synchronic teams, synchronic collaboration may still be seen as effective because they completed the task sooner. In Experiment 4, the conclusions of Experiment 1 are tested in conditions where teams of four complete four different problems in either diachronic or synchronic teams over the same 8 calendar hour time frame.
Two ways of allocating the same number of calendar hours toward multiple problems. Different people are different colors, and different projects are different shapes. In synchronic teams, team members work on each problem from beginning to end in lockstep. In diachronic teams, team members work on different problems at the same time, and collaborate over time.
Procedure. Experiment 4 is conducted like Experiment 1, except both teams now compete for 8 hours. Diachronic teams are now placed in the same room with three workstations so they can work all at the same time, though they are prevented from working on unassigned problems.
Research questions
Do the results extend to multiple problems? The predicted results for Experiment 4 are that diachronic teams will outperform synchronic teams across multiple problems.
Do teams get more efficient? Participants in Experiment 4 work on four different problems in a row. Do they get better at solving adaptive problems?
Summary
These experiments are designed to reveal the problem solving capacities of teams organized either diachronically or synchronically. The purpose of this investigation is not to prescribe certain team structures over others, since it seems likely that the most effective teams have elements of both team structures, but to outline the conditions for which diachronic collaboration is most effective. Diachronic collaboration is hypothesized to be most effective in solving problems that can be adapted in response to feedback. Since adaptive problem solving forms the basis for technological evolution, these results suggest that the methods and mechanisms that allow for diachronic collaboration might be critical to explaining the evolution of our species.
References
Basalla, G. (1989). The evolution of technology. Cambridge University Press.
For some preliminary evidence of the role of iteration in completing real world classification problems, see my analysis of classification competition leaderboards.↩