ACET, Inc. Blog

How does an evaluator decide what software packages to try and to take on for regular use? From database maintenance to note-taking to report production, evaluators use a wide variety of software packages in their daily work. Software makes handling large amounts of data and presenting them in interesting ways efficient and easy, and allows evaluators to share the information with clients in dynamic ways. There are many, many different packages to choose from, and it seems like there’s a new one appearing all the time. ACET evaluators were asked this week to talk about the factors that help them select new software solutions.

Cost. It’s an obvious factor, but the price of software does not necessarily prevent ACET evaluators from pursuing it as an option, internally or for a client. The functionality of the software was generally given greater weight by ACET staff rather than cost. In addition, when cost may be a concern there are many licensed and free, open source packages alike that ACET utilizes on a regular basis.

Learning, Training, and Sharing. New software packages take time to learn how to use, as well as teach to a client who may need the features. As a result, time needed to become proficient with software is a consideration for ACET when selecting new software. Everyone who needs access to the software to view, edit, add, or retrieve data (both inside and outside of the office) should be able to do so with a minimum amount of difficulty.

Usability. For more than one ACET evaluator, choosing between software packages came down to a simple question: “Does it do what I need it to do?” If a software package is similar to a program currently used in the office, or if there wasn’t a specific, preexisting need for another type of software package, assessing a new package wasn’t even a consideration.

What do you consider when looking at a new software package? We’d love to hear from you in the comments.

ACET Staff

Recent articles regarding teacher performance evaluation have shed light on the phenomenon of leniency bias. Also known as leniency error and performance or interview bias, leniency bias comes when an interviewer or manager inappropriately or unfairly rates an employee in a positive light. It also comes into play when jurors  [opens PDF] develop favorable opinions of a defendant in a court case outside of the facts presented.

With regard to evaluation, leniency bias has been described using the famous, fictional Minnesota town Lake Wobegon: everyone there is above average in some way. When assessing teacher effectiveness, as many as 98 percent of teachers were rated effective or better, regardless of how their students were performing, and relatively few (3 to 4 percent) were given the lowest possible effectiveness ratings and subsequently fired for poor performance. This comes despite calls for greater accountability from teachers and education administrators and claims that “5 to 10 percent” of teachers are ineffective. Outside of education, private companies with multiple tiers of performance tend to rate most employees in their top two echelons, above average and higher.

Why, when given the opportunity to identify ineffectiveness in a system and rectify the “problem” of ineffective teachers, does it happen so infrequently? In the example of teacher effectiveness, one reason is that the effectiveness (or lack thereof) changes depending on the measurement. At one point, Florida teachers were being evaluated half on student test scores and half on classroom evaluations, but the acceptable test score range has changed frequently in recent years, while the observation assessment at one point consisted of a single 20 minute observation in one year. Because of these shifting measurements, from year to year teachers can fluctuate between being labeled “effective” to “ineffective” and back again, rendering the designations moot.

Another factor that has been discussed are the risks that principals and other supervisors must take into account when determining effectiveness beyond performance. The demands on and for a teacher can differ widely depending on socio-economic and geographic impact, as does the likelihood of quickly replacing a teacher. As previously discussed on ACET’s blog, the state of Minnesota’s own desire to measure more complicated indicators of student growth and success has led to the adoption of its own standards. Principals who must contend with low turnover and a low supply of available teachers in the local job market risk a lengthy, expensive search for replacements in a field where up to 60 percent of professionals leave after their third year (per the Georgetown Public Policy Review). Rather than undertaking such searches, some principals may feel that it is better to stick with an ineffective but established teacher than expend resources on an unknown quantity that could potentially be even worse.

Another proposed reason for this leniency bias is that there is no bias at all, that there just isn’t “5 to 10 percent” worth of ineffective teachers to remove. Foundations and publications have reviewed the evaluations that produced so few identifications of ineffectiveness and found the resulting data, while not yet ready for mass consumption, conforms to their own suspicions of the true level of teacher ineffectiveness (3 to 4 percent, rather than the proposed 5 to 10).New evaluation systems continue to be developed to address these factors. However, it seems appropriate to conclude that as soon as a solid set of measurements has been decided on, it will be easier to determine if teacher effectiveness is the indicator of student success that it has been held up to be.

Mary

Last week, a group of American Evaluation Association (AEA) members released new results from a multi-year investigation into the state of evaluation among nonprofits nationwide. The story they tell about the relationship of American nonprofits and evaluation is very interesting. You can read the full report here. Among the highlights:

• Of the 546 nonprofit organizations surveyed, 90% had evaluated some aspect of their work in the previous year, but only 28% exhibited promising evaluation capacity or behaviors.

• All (100%) of organizations that engaged in evaluation used their findings primarily for internal reporting purposes (to funders, boards of directors, and stakeholders). Less than half used evaluation findings to report to clients (47%).

• A majority (70%) of surveyed nonprofits spent less than 5% of their organizational budgets on evaluation.

• Among a possible 10 priorities that included areas like financial management and communications, evaluation was ranked the second lowest by nonprofits. (The lowest ranked priority was research.)

At ACET, we understand the value of evaluation for nonprofits, not just because we are evaluators, but also because we have helped nonprofits build capacity and implement evaluation as a greater part of their success. We make this relationship more meaningful through:

Training and capacity building: We are happy to train and guide you and your staff through your evaluation, and we stay available throughout the project should you have any questions about the process. We want you to come away from the evaluation experience feeling confident and empowered to perform evaluation tasks on your own, if you so choose, and to have a full understanding of what evaluation is and is not about.

Involving staff at every step: We value your organization’s resources, talent, and input. That’s why we want to meet with you at the beginning and throughout the evaluation process to understand your goals, to get your feedback about every stage of the project, and to answer any questions you might have. The ability to steer the evaluation always remains in your control, and your voice will always be heard.

Assistance with reporting: You may have specific and detailed plans for communicating the data gathered during the evaluation process. Or, you may be unsure how to make your data available to your clients or stakeholders in the most effective and useful ways. At ACET, we are able to help you determine what format(s) will best meet your needs and goals, and to create professional, influential reporting materials for web and print – whether your audience is your community, your board, or the federal government.

We know these elements help nonprofit organizations achieve success – whether that be in the form of renewed or increased funding streams, a clarified vision for future program expansion, or answers to burning questions – because we have been instrumental to the success of many nonprofits. You can find out more about these successes by viewing the Projects page of our website. If you are a nonprofit decision maker unsure of how evaluation fits into your work, let us help you to explore the possibilities.

ACET Staff

The International Organisation for Cooperation in Evaluation (IOCE) is a network of regional and national evaluation societies and acts as a resource hub for evaluation professionals. One of these many resources is an interactive map showing organizations contributing to the IOCE as well as national organizations that have responded to their Voluntary Organizations of Professional Evaluators (VOPE) survey.

In the above map, every blue square represents public and private sectors, while the stars denote the larger regional organizations. When you think about evaluation internationally, where would you expect to see established evaluation societies? Only in “first world” nations? Only in nations with large international cities? I feel that this map challenges those conceptions. I am amazed at how many different evaluation societies there are all over the world, with memberships between the dozens and the hundreds, representing public and private sectors alike. From Nepal to Papua New Guinea to Paraguay, evaluators are truly part of an international community.

Mary

Recently, John Gargani shared a list of 5 words that evaluators use that can be misleading in reports. John’s blog inspired us to identify “pet peeves” we have found in evaluation work. Here’s our own list:

“Problem”

The use of the word “problem” in itself can elicit negative responses and put people on the defensive. When people are defensive, it can be difficult to engage in open dialogue and creative problem solving. Instead of using the word “problem,” Stella recommends using creative solutions to frame the issue to facilitate discussion. For example, Stella suggests using the word “challenge,” as in, “Participants identified some challenges in receiving quality health care…”

“Larger Population”

Joseph highlighted the overuse of questions about a “larger population,” as in, “how do our results compare to the larger population?” In general, evaluation results are obtained from a specific population within a specific context. Even results from “larger populations” are obtained from specific, albeit larger, populations within a specific context (e.g., Minnesotan 3rd through 8th grade students), and not from some generic “larger population.” In order to make valid comparisons, it’s important that evaluators always classify both their population of interest (e.g., South Minneapolis teens, between the ages of 13 and 17, from households with incomes of $25K or less) and the comparison group (e.g., a sample of Minneapolis teens from households with a variety of income levels). As often as possible, evaluators should not use generic “larger populations,” but should be as specific as possible about their comparison groups to enhance the validity and accuracy of their comparisons.

Describing the “Other” Category

Dan pointed out that, when analyzing qualitative data, there might be 5 common themes, but also an “other” theme for comments from a small number of people. When summarizing the results of the qualitative analysis it is important to talk about all of the themes, including describing what “other” indicates. Without describing what “other” indicates, the reader may have no idea as to what the writer is referring and may have to guess what “other” indicates. It is important that evaluators be as clear as possible when writing to ensure that the reader has a clear understanding of the data.

Pronouns

Kirsten’s pet peeve is overuse or abuse of pronouns. Pronoun overuse can detract from the clarity of writing and the overall readability of the document. From a grammatical standpoint, a pronoun always refers to the last used noun; if the last used noun isn’t what the writer intends to refer to, then a pronoun is inappropriate and the writing will not be as clear as the writer intended.

As a rule of thumb, Kirsten alternates noun-pronoun-noun-pronoun. She acknowledges that her method doesn’t always produce highly engaging prose, but it is clear!

Do you have any writing “pet peeves”? If so, please share in the comments below!

ACET Staff

Over the last few weeks, many of our clients have inquired about the availability of their test data from the Minnesota Department of Education (MDE) to meet reporting requirements. Here’s the most recent update from MDE’s Research and Assessment division:

“MDE is still evaluating the specific timelines for the release of the Reading and Mathematics MCA, MCA-Modified, and MTAS results, and we will provide one week’s notice before the results will be released. While we understand that not having a specific date makes it difficult for your planning purposes, we are holding some flexibility in the event we are able to complete our work and release results earlier than late September. Once a final determination is made, we will provide one week’s notice before results are released. We thank you for your patience as we continue work through the challenges created by the government shutdown.”(http://www.education.state.mn.us/MDE/Accountability_Programs/Assessment_and_Testing/index.html)]

We will keep you updated on projected timeline and new information when they become available.

Joseph

Please join us in welcoming our new Administrative Assistant, Kate Price. Prior to joining ACET, Kate provided support in multiple fields including outdoor recreation, volunteer coordination, and public schools. She also studied visual communications and completed her design internship in one of Minnesota’s largest park districts. Kate looks forward to furthering her career and life experience in the areas of communications, education, and social justice. In her spare time, she sings in a band, drags her boyfriend to an almost criminal amount of family get-togethers, and drinks a lot of coffee.

Do you have questions about program evaluation? Is there something specific you’d like to see covered on our blog? We encourage you to send us your questions or suggestions. Feel free to leave a comment on any blog post, or alternately, you can contact Cassie at cassie@acetinc.com or at 952-922-1811.

ACET will be attending the 2011 Public Affairs Student Association (PASA) Career Engagement Fair on Feb. 24 at the Hubert H. Humphrey School of Public Affairs at the University of Minnesota. We would love to talk with interested parties about our company and the field of evaluation. We will also be prescreening for positions at the fair. If you’re a graduate student or alumnus interested in a career in research and evaluation, please find our table – we would love to talk to you!

Cassie

Want to join our team in a leadership role? ACET is currently looking to hire a Senior Research Associate. Please click here to review the posting: Senior Research Associate