The many different approaches to the evaluation process are fascinating to examine, and this week we’d like to look at the Collaborative Outcomes Reporting approach: what is it, how is it used, and what does it uniquely contribute to evaluations?
Collaborative Outcomes Reporting is described as a combination of contribution analysis and Multiple Lines and Levels of Evidence (or MLLE), so it’s important to understand what those concepts are first. Contribution analysis is a technique by which stakeholders develop a story of evidence to support and improve a theory of change (i.e., why program activities lead to intended outcomes). Contribution analysis is not by itself a comprehensive evaluation approach. Multiple Lines and Levels of Evidence similarly seeks to establish a relationship between a program’s intervention and observed impacts, except that MLLE accomplishes this by convening a multi-disciplinary panel of experts to weigh a broad range of evidence. Collaborative Outcomes Reporting brings these concepts together by blending the MLLE’s expert review with data gathering and story building processes of contribution analysis. It’s good to note that the concept of “expert” is more expansive under Collaborative Outcomes Reporting – for example, an evaluation of community programming may have a citizen’s jury as its expert panel rather than published individuals with advanced degrees.
There are four steps to the Collaborative Outcomes Reporting approach.
- In the planning and design phase, the theory of change is developed, and the questions to be answered by the evaluation process are identified.
- In the data collection phase, evidence is gathered to support and build up the theory of change. This data collection may take the form of quantitative data, such as test scores, or qualitative data, such as interviews (called social inquiry), or both.
- In the outcomes phase, the expert panel is convened and presented with the evidence gathered in the Data Collection phase. The panel is then asked to review the programs impact with regard to its goals and the gathered evidence.
- Finally, in the reporting phase, key findings, instances of significant change, implications, challenges, and recommendations are pulled together and presented to key stakeholders, staff, and the community. This can take place at an event, called a summit workshop, or be combined in a report, or both.
Under the right circumstances, the Collaborative Outcomes Reporting approach can be very beneficial. The process is highly participatory, frequently placing key stakeholders, program staff, or community members at the forefront of the decision making and data analysis process while the evaluators facilitate discussion or act as managers. The report structure is easy to understand and aligns with the common desires of stakeholder groups such as boards of directors. The simplicity of the process and the focus of the approach can be seen as an advantage and a possible disadvantage, depending on the complexity of the program. The approach is also particularly useful when the program has emergent or complicated outcomes that are not well defined at its inception.