ACET, Inc. Blog

Last week, a group of American Evaluation Association (AEA) members released new results from a multi-year investigation into the state of evaluation among nonprofits nationwide. The story they tell about the relationship of American nonprofits and evaluation is very interesting. You can read the full report here. Among the highlights:

• Of the 546 nonprofit organizations surveyed, 90% had evaluated some aspect of their work in the previous year, but only 28% exhibited promising evaluation capacity or behaviors.

• All (100%) of organizations that engaged in evaluation used their findings primarily for internal reporting purposes (to funders, boards of directors, and stakeholders). Less than half used evaluation findings to report to clients (47%).

• A majority (70%) of surveyed nonprofits spent less than 5% of their organizational budgets on evaluation.

• Among a possible 10 priorities that included areas like financial management and communications, evaluation was ranked the second lowest by nonprofits. (The lowest ranked priority was research.)

At ACET, we understand the value of evaluation for nonprofits, not just because we are evaluators, but also because we have helped nonprofits build capacity and implement evaluation as a greater part of their success. We make this relationship more meaningful through:

Training and capacity building: We are happy to train and guide you and your staff through your evaluation, and we stay available throughout the project should you have any questions about the process. We want you to come away from the evaluation experience feeling confident and empowered to perform evaluation tasks on your own, if you so choose, and to have a full understanding of what evaluation is and is not about.

Involving staff at every step: We value your organization’s resources, talent, and input. That’s why we want to meet with you at the beginning and throughout the evaluation process to understand your goals, to get your feedback about every stage of the project, and to answer any questions you might have. The ability to steer the evaluation always remains in your control, and your voice will always be heard.

Assistance with reporting: You may have specific and detailed plans for communicating the data gathered during the evaluation process. Or, you may be unsure how to make your data available to your clients or stakeholders in the most effective and useful ways. At ACET, we are able to help you determine what format(s) will best meet your needs and goals, and to create professional, influential reporting materials for web and print – whether your audience is your community, your board, or the federal government.

We know these elements help nonprofit organizations achieve success – whether that be in the form of renewed or increased funding streams, a clarified vision for future program expansion, or answers to burning questions – because we have been instrumental to the success of many nonprofits. You can find out more about these successes by viewing the Projects page of our website. If you are a nonprofit decision maker unsure of how evaluation fits into your work, let us help you to explore the possibilities.

ACET Staff

When ACET staff met to debrief about the recent AEA Conference in Minneapolis, they had lots of interesting and useful information that they gleaned from days of attending presentations and discussions to share. Here are some comments.

Kirsten: I gravitated towards presentations on data visualization and reporting and sessions in which a skill or technique was demonstrated. In one of the sessions, Simon Geletta (Des Moines University) showed how to build information dashboards. The software was surprisingly easy to use, primarily because it is based on a knowledge and understanding of Excel. Also, if you ever have a chance to watch an Ignite presentation, you may want to do it. Ignite speakers have 5 minutes to talk and 20 presentation slides. The catch? The slides automatically forward every 15 seconds, whether or not the speaker is ready. The result is often a polished, focused summary of the presenter’s work or position. They were inspiring to watch!

Stella: I went to a presentation about college access programs that gave me lots to think about, like the different rubrics used to measure engineering programs. I also went to a session focused on environmental impact that expanded my horizons on what indicators to include for future opportunities. I was also impressed by the work of the University of Minnesota Extension’s cohort model for building evaluation capacity.

Joseph: One presentation I thought was particularly interesting was about evaluation in a for-profit, corporate environment. It ended up being more different [from nonprofit evaluation] than I could have previously imagined. The focus is more on process over outcome, with the understanding that process will lead to outcomes (new business, repeat business, etc.). I did learn some good takeaway strategies that furthered my process evaluation knowledge and skills that will be useful for any sector.

Dan: I found a couple of presentations particularly interesting. One was about using email and postal surveys (or survey invitations) together to maximize response rates, and it went through the different combinations [the presenter] used to deliver the survey (or invite people to complete the survey). Another session I attended was about the analysis of quantitative data with different software packages, and there were some that I knew of, but others I didn’t. I found the functionality of each different software package to be of interest.

ACET Staff

A large part of the job of an evaluator is capacity building, or instructing and involving clients in the activity of evaluation and the use of their subsequent results, even sometimes as an evaluation is taking place. Effective capacity building is a regular topic among evaluators, and there are many methods that may be employed depending on the needs of a particular client. Today I’d like to explore one such method, empowerment evaluation.

Empowerment evaluation, or EE, involves the use of reflection and self-evaluation on the part of clients to help them make decisions and improve and grow their projects even after the evaluation is over.

For one approach to empowerment evaluation, here is a 3-step process:

1. Establish the mission. What does the client want the project or their work to be about going forward?
2. Take stock of the project. Based on the vision established, where is the client currently in achieving the program’s goals?
3. Plan for the future. What benchmarks or strategies can be put into place to get or keep the client on track for the program’s goals?

An evaluator assists clients through empowerment evaluations in many ways. Traditional evaluation tools, such as surveys and interviews, may be used to take initial stock of the client’s current situation as well as to gauge progress later on. The evaluator may act as a coach throughout the process. New technologies may be recommended for introduction into the communication or data collections processes of a client’s organization.

The needs of any client include and many contain a wide variety of factors, issues, and challenges, and as such no exact system can be said to be the right fit for all clients, but the potential of empowerment evaluation is said to be limited only by one’s own imagination.

Mary

Check out this video for what not to do in faciliating a dialogue! At just over a minute, this is a short and comical clip that teaches us a few lessons on what to avoid when conducting interviews and/or focus groups.

Notice the kitty’s pointing finger, negative facial expressions, threats of tattling, unnecessary directness and belittling. Who knew such an adorable critter could be so off-putting? That poor puppy didn’t know what to do with himself!

What are some lessons you’ve learned about facilitation?

We often get requests to provide feedback on surveys. As we review the surveys, we tend to see three challenges occur quite frequently. I’ve written this blog to offer suggestions for addressing each of the challenges.

1. Double-Barreled Questions. In a double-barreled question, two concepts or ideas are included in one survey question. For example, the three survey items below are all double-barreled questions:
○        “How much of your monthly income is spent on restaurant dining and entertainment?”
○        “How often do you volunteer your time or make charitable donations?”
○        “How often do you consume beer and smoke cigars?”

But how does a respondent answer a double-barreled question? For example, if a family spent no money on restaurant dining but 10% of their income on entertainment, how should they respond? 0%? 10%? Average the two and respond 5%? Usually survey respondents do provide an answer, but they often utilize an individualized strategy for finding a response, which can be challenging for program staff to interpret.

Double-barreled questions can be addressed by limiting each survey item to one – and only one – concept. Below is a double-barreled question followed by revisions that eliminate the double-barrel:

2. Imbalanced Response Options. Imbalanced response options refer to a set of response choices that do not cover the range of possible choices. An imbalanced response scale is problematic because it limits respondents’ responses to only one end of a scale and may produce inaccurate results. For example, if a respondent wants to answer “strongly disagree” to a question but that option is not available, the survey is not capturing the respondent’s perspective. Using a set of balanced response options ensures that respondents’ answers can be accurately captured. A very good resource for a variety of balanced response scales can be found here: http://www.gifted.uconn.edu/siegle/research/Instrument%20Reliability%20and%20Validity/Likert.html

Below are some examples of imbalanced response options with improvements to the scale:

3. Overlapping Response Options. Overlapping response options simply means that there is some overlap in the response choices available to the respondent. For example, the following survey item has overlapping response options:

How many books did you read in the last month?
0 or 1 book
1 or 2 books
2 or 3 books
3 or more books

Overlapping response options are problematic because they can confuse the survey respondent. If you read two books in the last month, which option would you select: “1 or 2 books” or “2 or 3 books”? In addition, questions with overlapping response options are a challenge to interpret. How do you know how many books were read if the response options overlap? Overlapping response options can be addressed very easily by carefully constructing the response option list. Simply revise and edit the list so that there is no overlap between the response choices.

If you have any questions about the above challenges to designing a survey, please feel free to leave a comment or email me at Kirsten@acetinc.com. Please also contact any of us at ACET if you have questions about survey design in general.

Kirsten

At ACET, we strongly believe in “capacity building” – but what does that mean? Basically, we believe in creating and implementing evaluation plans that involve our clients. Our clients have input in every step of the evaluation process, which enables us to find the plan that best suits the needs of their program or organization. In turn, they learn how to use the evaluation results to improve programming. Since capacity building is so important to us, we work hard to ensure that we maintain open and transparent communication with our clients.

Stella learned of a video while at the American Evaluation Association conference that summed up precisely what NOT to do when communicating with your clients.

http://www.youtube.com/watch?v=AukrpAoAYY0

The video is amusing, but it brings up some essential key points.

“What does this mean?” Use clear language that clients can understand. Those test results may be second nature to you, but like the man in the video, your client may have no idea what those numbers actually mean.

“What does a score of 35 tell me about my patient?” In our experience, clients don’t want random numbers to throw around; they want solid information that they can use to improve their programming. Be sure to clearly delineate what their results mean and answer any questions they may have.

“I used the scale you developed to measure my patient.” Your client trusts that you have expertise about evaluation – after all, they’re using products you likely designed! – but being able to communicate that knowledge is essential in maintaining the relationship.

The end of the video is also telling. Despite the researcher’s supposed credentials, the client doesn’t want to deal with her anymore. Clients, too, may want to stop working with someone they feel isn’t right for their organization, even if that person or company has years of accolades.

Do you have any thoughts to share about communicating with clients? We’d love to hear them!