Potential alternative approaches to evaluating knowledge management (KM) program performance

A question often raised in knowledge management (KM) forums is how to show return on investment in KM. Most recently, a member of the SIKM Leaders Community asked:

The question which is always asked on KM performance is, what is the benefit. Change in culture, improvement in collaborating, creation of assets, etc. is not an answer to this. since this is just an output but not measure of benefit/success. Can someone share, how we can correlate KM with business benefits OR how can we show return on investment in KM. Thank you.

Demonstrating return on investment has also been an issue for the natural resource management (NRM) sector, where I’ve worked for much of my career. Indeed, a 2008 performance audit report from the Australian National Audit Office (ANAO) examining the regional delivery of nearly $3 billion in funds through the Natural Heritage Trust (NHT) states that¹:

Performance monitoring, evaluation and reporting are essential for determining the extent to which agencies’ outputs and administered items contribute to the achievement of the program outcomes. ANAO audits in 1996–97, 2000–01 and again in 2004–05 found weaknesses in the monitoring and reporting of the performance of the NHT. In summary there was no effective outcomes reporting.

Aligning with the question above to the SIKM community, Australia’s state and territory agencies and 56 regional NRM bodies could clearly demonstrate outputs from the investment – that is, activities undertaken such as tree planting, environmental weed removal, and training events – but much less so the outcomes achieved through these activities.

In response, the NRM sector reached out to the evaluation community for advice, and as stated in the 2008 ANAO report, then trialled and implemented program logic and performance story reporting. At that time, the NRM sector had already reached out to the KM community for advice, establishing the Knowledge for Regional NRM Program which I worked on from 2006-8. The activities of the Knowledge for Regional NRM Program included support for program logic and performance story reporting.

Program logic and performance story reporting are introduced below, together with two other approaches that are also increasingly being used in the NRM sector and beyond – most significant change (MSC) and multiple lines and levels of evidence (MLLE).

Just as the NRM sector has benefitted from the knowledge of the evaluation and KM communities, the KM community can potentially benefit from the knowledge of the evaluation and NRM communities. The numerous examples of research and case studies from the NRM sector in the taking responsibility for complexity series already demonstrate how the KM community can learn from the NRM sector’s extensive experience and expertise in managing in the face of complexity.

Program logic and performance story reports

As well as being used in the NRM sector, program logic and performance story reports are used widely in evaluating the performance of programs in both government and the private sector.

A program logic model sets out the resources and activities that comprise the program, and the outputs and outcomes that are expected to result from them. The logic model is both a tool for program planning and for evaluation because it draws out the relationships between resources, activities, and outcomes. It is also a very useful tool for communication and engagement, including communicating program successes to senior management.

The following resources provide comprehensive guidance in how to develop a program logic, and then based on the logic, tell a performance story for a program:

• Logic Models: A Tool for Telling Your Program’s Performance Story² – paper published in the journal Evaluation and Program Planning

• Exploring Program Logic – video from the New South Wales (NSW) Ministry of Health (can also be viewed at the top of this page)

• Developing and Using Program Logic: A Guide³ – comprehensive guide from the New South Wales (NSW) Ministry of Health

• How to develop a program logic for planning and evaluation – resources from the Australian Government Institute of Family Studies, including a program logic template with guidance, an example of a completed program logic, and a blank template

• Developing and Using Program Logic in Natural Resource Management, User Guide⁴ – comprehensive and easy to follow user guide developed for the NRM sector (designed for use in conjunction with the companion user guide immediately below)

• Developing a Performance Story Report, User Guide⁵ – comprehensive and easy to follow user guide developed for the NRM sector (designed for use in conjunction with the companion user guide immediately above)

• Outback to the Ocean Performance Story Report⁶ – an example of program logic and performance story reporting from the NRM sector

• Mount Lofty Ranges Southern Emu-wren and Fleurieu Peninsula Swamps Recovery Program Performance Story Report⁷ – an example of program logic and performance story reporting from the NRM sector.

Most significant change (MSC)

The most significant change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data, and it contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

The following resources introduce the MSC technique and provide guidance in its use:

• The Most Significant Change Technique⁸ – an introduction to the MSC technique, produced as part of the Knowledge Solutions series from the Asian Development Bank (ADB)

• The Most Significant Change Technique: A Guide to its Use⁹ – a comprehensive guide to the MSC technique and its use.

Multiple lines and levels of evidence (MLLE)

Establishing cause and effect relationships is challenging when seeking to evaluate the performance of complex interventions. As I’ve discussed in a previous RealKM Magazine article, the multiple lines and levels of evidence (MLLE) approach is very useful in this regard. It involves bringing together different types of evidence, and systematically analysing the strength of the causal argument linking an intervention or a cause to its effects.

The following resources introduce the MLLE approach and provide guidance in its use:

• Using multiple lines and levels of evidence (MLLE) to generate knowledge for causality inference – RealKM Magazine article

• Learning from the evidence about evidence-based policy¹⁰ (see page 204) – discussion of the MLLE approach in a presentation given as part of the Australian Productivity Commission 2009 roundtable on the topic of “Strengthening Evidence-Based Policy in the Australian Federation”

• Causal Criteria Methods Manual – Methods for applying the multiple lines and levels of evidence (MLLE) approach for addressing questions of causality¹¹ – comprehensive guide on how to use the MLLE approach

• Evaluation Design – brief guide from Environment NSW on how to use the MLLE approach

• Recording assessment using MLLE criteria template – template from Environment NSW to record all the lines of evidence available for each evaluation question, for use with the MLLE guides directly above

• Final Evaluation of the Improved Global Governance for Hunger Reduction Programme¹² (see page 6) – an example of the application of the MLLE approach

• Review of UN SWAP Evaluation Performance Indicator Reporting¹³ (see page 7) – an example of the application of the MLLE approach

• Determining and managing environmental flows for the Shoalhaven River¹⁴ (see page 50) – an example of the application of the MLLE approach

• Review of the 1999 ACT Environmental Flow Guidelines¹⁵ (see Appendix D) – an example of the application of the MLLE approach.

References:

Also published on Medium.