ACET, Inc. Blog

How does an evaluator decide what software packages to try and to take on for regular use? From database maintenance to note-taking to report production, evaluators use a wide variety of software packages in their daily work. Software makes handling large amounts of data and presenting them in interesting ways efficient and easy, and allows evaluators to share the information with clients in dynamic ways. There are many, many different packages to choose from, and it seems like there’s a new one appearing all the time. ACET evaluators were asked this week to talk about the factors that help them select new software solutions.

Cost. It’s an obvious factor, but the price of software does not necessarily prevent ACET evaluators from pursuing it as an option, internally or for a client. The functionality of the software was generally given greater weight by ACET staff rather than cost. In addition, when cost may be a concern there are many licensed and free, open source packages alike that ACET utilizes on a regular basis.

Learning, Training, and Sharing. New software packages take time to learn how to use, as well as teach to a client who may need the features. As a result, time needed to become proficient with software is a consideration for ACET when selecting new software. Everyone who needs access to the software to view, edit, add, or retrieve data (both inside and outside of the office) should be able to do so with a minimum amount of difficulty.

Usability. For more than one ACET evaluator, choosing between software packages came down to a simple question: “Does it do what I need it to do?” If a software package is similar to a program currently used in the office, or if there wasn’t a specific, preexisting need for another type of software package, assessing a new package wasn’t even a consideration.

What do you consider when looking at a new software package? We’d love to hear from you in the comments.

ACET Staff

Recent articles regarding teacher performance evaluation have shed light on the phenomenon of leniency bias. Also known as leniency error and performance or interview bias, leniency bias comes when an interviewer or manager inappropriately or unfairly rates an employee in a positive light. It also comes into play when jurors  [opens PDF] develop favorable opinions of a defendant in a court case outside of the facts presented.

With regard to evaluation, leniency bias has been described using the famous, fictional Minnesota town Lake Wobegon: everyone there is above average in some way. When assessing teacher effectiveness, as many as 98 percent of teachers were rated effective or better, regardless of how their students were performing, and relatively few (3 to 4 percent) were given the lowest possible effectiveness ratings and subsequently fired for poor performance. This comes despite calls for greater accountability from teachers and education administrators and claims that “5 to 10 percent” of teachers are ineffective. Outside of education, private companies with multiple tiers of performance tend to rate most employees in their top two echelons, above average and higher.

Why, when given the opportunity to identify ineffectiveness in a system and rectify the “problem” of ineffective teachers, does it happen so infrequently? In the example of teacher effectiveness, one reason is that the effectiveness (or lack thereof) changes depending on the measurement. At one point, Florida teachers were being evaluated half on student test scores and half on classroom evaluations, but the acceptable test score range has changed frequently in recent years, while the observation assessment at one point consisted of a single 20 minute observation in one year. Because of these shifting measurements, from year to year teachers can fluctuate between being labeled “effective” to “ineffective” and back again, rendering the designations moot.

Another factor that has been discussed are the risks that principals and other supervisors must take into account when determining effectiveness beyond performance. The demands on and for a teacher can differ widely depending on socio-economic and geographic impact, as does the likelihood of quickly replacing a teacher. As previously discussed on ACET’s blog, the state of Minnesota’s own desire to measure more complicated indicators of student growth and success has led to the adoption of its own standards. Principals who must contend with low turnover and a low supply of available teachers in the local job market risk a lengthy, expensive search for replacements in a field where up to 60 percent of professionals leave after their third year (per the Georgetown Public Policy Review). Rather than undertaking such searches, some principals may feel that it is better to stick with an ineffective but established teacher than expend resources on an unknown quantity that could potentially be even worse.

Another proposed reason for this leniency bias is that there is no bias at all, that there just isn’t “5 to 10 percent” worth of ineffective teachers to remove. Foundations and publications have reviewed the evaluations that produced so few identifications of ineffectiveness and found the resulting data, while not yet ready for mass consumption, conforms to their own suspicions of the true level of teacher ineffectiveness (3 to 4 percent, rather than the proposed 5 to 10).New evaluation systems continue to be developed to address these factors. However, it seems appropriate to conclude that as soon as a solid set of measurements has been decided on, it will be easier to determine if teacher effectiveness is the indicator of student success that it has been held up to be.

Mary

This week, we checked in with ACET staff again to find out what we’ve been reading.

Joseph: Why Does the World Exist by Jim Holt: This book is a series of interviews with physicists and philosophers on the old philosophical question of “why is there something rather than nothing?” Put another way, “why is there existence rather than nothing?” Positions vary on this from the question being irrelevant (the state of nothing being impossible) to the principals of quantum mechanics requiring there be something to something being ethically better than nothing. It was a fascinating read (I just finished!) and would appeal to anyone who has ever pondered this or is awe struck by the state of existence.

Kirsten: I recently finished reading Hunting Eichmann by Neal Bascomb. The first half of the book describes Adoph Eichmann’s life during the final days of the war, surviving under false identities immediately after the war, and his escape to and life in Argentina. For example, did you know that Eichmann was arrested and imprisoned in allied POW camps twice, and managed to escape both times? The second half of the book explains how the Israeli government found Eichmann and planned for his capture and transport to Israel, the trial, and final judgment. If you are interested in World War II history, you might want to check it out!

Dan: Lately I’ve been reading up on my professional development resources in preparation to present at the ADARA (formerly known as the American Deafness and Rehabilitation Association) 2013 conference. I’ve become engrossed with articles on survey design and development best practices.

Elizabeth: I am currently reading Women Who Run with the Wolves by Clarissa Pinkola Estes. Estes discusses the instinctual nature of women, which is illustrated through the book with bits of folklore, myths, and fairy tales. Estes believes that women’s instinctual nature has been repressed, and in order for women to become whole, they must embark on a path of self-discovery to examine their preferences and shortcomings, especially as these traits relate to societal norms and expectations. I haven’t gotten far enough into the narrative to really issue an opinion yet, but I enjoy the storytelling of the author and the examination of feminine myths.

Mary: Once again, I’m reading a graphic novel – this time it’s Volume 2 of Genshiken Second Season by Kio Shimoku. Short for Gendai Shikaku Bunka Kenkyuukai (or, the Society for the Study of Modern Visual Culture), Genshiken is a fictional collegiate student club that overlaps other, more respectable clubs for video games, animation, and comics by focusing on the fan culture for those mediums, especially the production of fan works. The new characters introduced at the beginning of this new series remain true to the depth, humor, and humanity of the first series, but I think it’s still taking off. The new central character seems to be an underclassman named Hato, a boy using media and fan works to navigate personal issues of gender, identity, and presentation.

Stella: I just started reading Now, Discover Your Strengths by Marcus Buckingham and Donald O. Clifton, PhD. It was recommended by a colleague of mine.

What have you been reading lately? Join the discussion in the comments.

ACET Staff

Last week, a group of American Evaluation Association (AEA) members released new results from a multi-year investigation into the state of evaluation among nonprofits nationwide. The story they tell about the relationship of American nonprofits and evaluation is very interesting. You can read the full report here. Among the highlights:

• Of the 546 nonprofit organizations surveyed, 90% had evaluated some aspect of their work in the previous year, but only 28% exhibited promising evaluation capacity or behaviors.

• All (100%) of organizations that engaged in evaluation used their findings primarily for internal reporting purposes (to funders, boards of directors, and stakeholders). Less than half used evaluation findings to report to clients (47%).

• A majority (70%) of surveyed nonprofits spent less than 5% of their organizational budgets on evaluation.

• Among a possible 10 priorities that included areas like financial management and communications, evaluation was ranked the second lowest by nonprofits. (The lowest ranked priority was research.)

At ACET, we understand the value of evaluation for nonprofits, not just because we are evaluators, but also because we have helped nonprofits build capacity and implement evaluation as a greater part of their success. We make this relationship more meaningful through:

Training and capacity building: We are happy to train and guide you and your staff through your evaluation, and we stay available throughout the project should you have any questions about the process. We want you to come away from the evaluation experience feeling confident and empowered to perform evaluation tasks on your own, if you so choose, and to have a full understanding of what evaluation is and is not about.

Involving staff at every step: We value your organization’s resources, talent, and input. That’s why we want to meet with you at the beginning and throughout the evaluation process to understand your goals, to get your feedback about every stage of the project, and to answer any questions you might have. The ability to steer the evaluation always remains in your control, and your voice will always be heard.

Assistance with reporting: You may have specific and detailed plans for communicating the data gathered during the evaluation process. Or, you may be unsure how to make your data available to your clients or stakeholders in the most effective and useful ways. At ACET, we are able to help you determine what format(s) will best meet your needs and goals, and to create professional, influential reporting materials for web and print – whether your audience is your community, your board, or the federal government.

We know these elements help nonprofit organizations achieve success – whether that be in the form of renewed or increased funding streams, a clarified vision for future program expansion, or answers to burning questions – because we have been instrumental to the success of many nonprofits. You can find out more about these successes by viewing the Projects page of our website. If you are a nonprofit decision maker unsure of how evaluation fits into your work, let us help you to explore the possibilities.

ACET Staff

As evaluators, ACET staff understand the importance of creating or selecting a measurement tool that constitutes the right fit for the client and the data to be measured. The Multiple Measurements Rating (MMR), implemented by the State of Minnesota last year following its No Child Left Behind waiver, is an attempt at just such a measurement tool. Recognizing the need for assessments with a focus beyond test proficiency, the MMR evaluates schools in four areas: achievement gap reduction, growth, and graduation rate, in addition to proficiency.

An article at MinnPost this week illustrated why different approaches to measurement are needed by focusing on one of Minnesota’s many charter schools that serve the unique needs of at risk youth, or those otherwise unable to thrive in the traditional school system. By the measurement standards of No Child Left Behind, these schools are consistently underperforming, but the concern of charter school and state legislature officials is that the federal measurements do not capture progress made towards standards but only whether or not students achieved standards.

Nationwide, states have expressed that waivers have given them the freedom to develop their own assessment tools as they see fit. However, others have expressed frustration that there no longer seems to be a unified national metric for school and student success, as so many states have opted out of No Child Left Behind and its measurement requirements. Is it possible to have measurement tools that satisfy these micro and macro needs?

Mary

The International Organisation for Cooperation in Evaluation (IOCE) is a network of regional and national evaluation societies and acts as a resource hub for evaluation professionals. One of these many resources is an interactive map showing organizations contributing to the IOCE as well as national organizations that have responded to their Voluntary Organizations of Professional Evaluators (VOPE) survey.

In the above map, every blue square represents public and private sectors, while the stars denote the larger regional organizations. When you think about evaluation internationally, where would you expect to see established evaluation societies? Only in “first world” nations? Only in nations with large international cities? I feel that this map challenges those conceptions. I am amazed at how many different evaluation societies there are all over the world, with memberships between the dozens and the hundreds, representing public and private sectors alike. From Nepal to Papua New Guinea to Paraguay, evaluators are truly part of an international community.

Mary

Just in time for Valentine’s Day, ACET’s evaluators were asked to share what they love about being evaluator, or what makes them passionate about evaluation. Here’s what they had to say:

Stella: What I like about evaluation is that I always get opportunities to be creative. You work with the client and customize the evaluation in a new way every time – it’s never a mundane process. There’s always something new to problem solve and to explore.

Joseph: I am passionate about presenting usable results to clients. One piece in the cycle of evaluation is to report to clients, but I enjoy thinking about how to best derive results that are most useful to clients and maximize usability. This thinking begins when I first learn about the client, and doesn’t end until I think we have squeezed all the useful information we can out of the available data.

Dan: One thing I love about being an evaluator is when I see the evaluation results being used. They could be used for program improvement, organization reporting requirements, or just to increase someone’s knowledge in a certain area. When I see the results being used, it makes me feel that I have effectively performed my job as an evaluator.

Elizabeth: What I love most about being an evaluator is working with clients to improve their operations or programming. Many people are unfamiliar with evaluation, and some people are nervous about the whole process. I love getting to know our clients and working with them to tailor our evaluation services to ensure the results will be useful for all involved. I think the close relationship that we at ACET forge with our clients can help alleviate some of the stress and anxiety associated with evaluation, and there’s nothing I like better than when clients get really excited about upcoming evaluation activities, and get really into planning the evaluation.

Kirsten: One of the things I love most about being an evaluator is problem solving. I love being faced with new and interesting challenges and finding creative solutions. It might be finding cost-efficient options for a client’s data collection, or the best way to present evaluation findings in reports, but each day is a new, wonderful opportunity to solve problems.

What do you love about being an evaluator, or about evaluation? Let us know in the comments.

ACET Staff

In these first few days of 2013, the staff of ACET was asked to share one thing they would like to do to become better evaluators. Here are some of their responses.

Dan: To become a better evaluator, I would like to participate in an upcoming conference. Participation in a conference will not only help improve my verbal and written skills, but will also help increase my overall professional development and networking capabilities.

Joseph: I would like to share more of my work with the community. We do some unique evaluation projects, and our clients show some interesting results and I think the evaluation community could benefit from our experiences.

Stella: This year, I would like to take my quantitative visuals to a new level by utilizing more of my background in art.

What are some things that you would like to do to become better at what you do in 2013? Let us know in the comments.

ACET Staff

You don’t need to search very hard to find writing on the Internet that extols the importance of thank you notes, but frequently it is limited to the realms of job interviews and gift giving. Handwritten tokens of thanks can be appropriate for events great and small that go beyond these. Everyone appreciates receiving mail with a personal touch, and it has a permanence that e-mail and telephone calls do not. Consider writing these people thank you notes during the evaluation process:

The Volunteer Stakeholder: Send thank you notes to community members who attend advisory group sessions or other individuals who volunteer their time and expertise to assist with evaluation efforts.

The Administrative Assistant: Send thank you notes to administrative staff whose hands touch the evaluation as it progresses (e.g., those distribute surveys, make copies, and package data to be sent back to the evaluator).

The Colleague: Send thank you notes to your colleagues who generously offer advice. It can be quick and easy to reach out to a colleague during an evaluation via e-mail for advice about a difficult project, but less easy to remember that person’s contribution after all is said and done.

Evaluators, who are some other people that you remember in your thank you notes?

Mary

Being an effective writer is an important skill for an evaluator. It takes time and practice. One of my favorite books on effective writing was written by Morris, Fitz-Gibbon, and Freeman, titled “How to Communicate Evaluation Findings.” Only 92 pages long, this book is indispensable at helping evaluators to communicate findings in a succinct way and is full of creative examples and advice. Here’s one of my favorite parts, regarding the choice of the right form for communicating findings to a given audience (page 31):

You are eager, perhaps, to show the audience the clarity and brilliant complexity of the evaluation plan, the quick-thinking administrative maneuvers that saved the evaluation from disaster, or the mathematical genius so obviously guiding the analyses. DO NOT DO IT! You will bore the audience to death and render the findings indigestible.

This excerpt sums up for me the beauty and the utility of this book, understanding those things that evaluators value in reports, and reminding us that they aren’t the same things that stakeholders and other audience groups value.

Stella