Critical Eye: Bird Island experiment claims to prove usefulness of KM

Critical Eye is a semi-regular feature where RealKM analyses and discusses the methodology and science behind claims made in publications.

In this column we are examining Knoco’s 2009 Bird Island case study, which involves teams “creating the tallest tower possible, provided that it passes a couple of stability tests” over two attempts, using teams and best practice learning to improve results in the second outcome, as an example of the effectiveness of KM techniques. The conclusion of this study states:

Capturing and sharing knowledge, and storing it in a regularly updated knowledge asset, has not only improved the performance of every team that has ever been through the simulation, it has also resulted in continuous innovation, design evolution and performance improvement over the past 10 years … If your senior managers need to be convinced that KM can improve performance and add value, show them this data. Or even better – get them to take part in the workshop, and to experience for themselves the empowerment and competence that comes from a full state of knowledge, and the performance that can result from this.

In short, the study appears to be a ‘golden bullet’ for convincing recalcitrant managers that KM is real and worth adopting. There is no doubt that the methodology is clever and the results convincing. The basic steps are:

1. Teams are told the rules and attempt to build a tower from first principles.
2. They are asked to re-estimate the height they would achieve on a second build of the tower three times: (a) following an after-action review, (b) after reviewing the methods used by a different team, and (c) after reviewing the practices of the best ever previous participant. Each time estimates tend to increase as people learn about practices better than those previously discovered.
3. Teams build the tower a second time using the knowledge gathered, generally achieving a height far superior to their first effort.

The final height of the tower build steadily increases on average over time and it appears pretty clear that teams are innovating based on the last, best known effort:

Unfortunately, there are two significant weaknesses with the study that limit its effectiveness as a robust study:

1. The comparative effectiveness of alternative learning techniques are not tested
2. The net productivity gain of this approach is not evaluated

In relation to the first weakness, if we assume that in all cases two towers will be built by participants, a range of alternate methodologies are possible such as:

• Build, build (no review stage)
• Build, after action review, build (no reference to best practice)
• Review best practice, build, build
• Review best practice, build, after action review, build

Without testing these alternatives, it is difficult to establish the relative gains from experience, peer learning, and access to reference material.

And in relation to productivity, in the real world it is not sufficient to ask whether knowledge is “higher quality” or not.

The Bird Island experiment does prove pretty conclusively that it is possible to improve performance in a fixed scenario over time with access to good historical knowledge. But “higher performance” cannot be simply equated with a “better” or “more productive” outcome.

In the Bird Island scenario, it may be that creating a “higher performance” bird-catching net takes three times the resources once the long-term costs of capturing, retaining, and updating the knowledge on how to build that net is accounted for, but only catches twice the number of birds. If this is true, then productivity is maximised by building three smaller nets, not by spending the time on the knowledge generation and retention activities.

There is a temptation for knowledge management to focus on improved performance. But fact, higher quality organisational knowledge is not an unambiguously positive goal. In a systems sense, a “better” outcome may result from objectively worse knowledge where the cost of attaining and maintaining knowledge is too great. The correct focus on knowledge will depend upon an appropriate assessment of risk and value unlocked in the full organisational context.

To sum up, it is clear that the Bird Island study demonstrates the potential of KM techniques to improve organisational performance. But it is not a slam dunk at proving that KM is a net positive for organisational productivity. The Bird Island study is, in reality, primarily a sales tool and it is likely to be perfectly adequate for that purpose. But given the current cynicism around KM’s ability to improve productivity, we need to be careful about over-enthusiastically promoting an approach for (expensively) improving performance.

Source: Knoco