ACET, Inc. Blog

Last week, a group of American Evaluation Association (AEA) members released new results from a multi-year investigation into the state of evaluation among nonprofits nationwide. The story they tell about the relationship of American nonprofits and evaluation is very interesting. You can read the full report here. Among the highlights:

• Of the 546 nonprofit organizations surveyed, 90% had evaluated some aspect of their work in the previous year, but only 28% exhibited promising evaluation capacity or behaviors.

• All (100%) of organizations that engaged in evaluation used their findings primarily for internal reporting purposes (to funders, boards of directors, and stakeholders). Less than half used evaluation findings to report to clients (47%).

• A majority (70%) of surveyed nonprofits spent less than 5% of their organizational budgets on evaluation.

• Among a possible 10 priorities that included areas like financial management and communications, evaluation was ranked the second lowest by nonprofits. (The lowest ranked priority was research.)

At ACET, we understand the value of evaluation for nonprofits, not just because we are evaluators, but also because we have helped nonprofits build capacity and implement evaluation as a greater part of their success. We make this relationship more meaningful through:

Training and capacity building: We are happy to train and guide you and your staff through your evaluation, and we stay available throughout the project should you have any questions about the process. We want you to come away from the evaluation experience feeling confident and empowered to perform evaluation tasks on your own, if you so choose, and to have a full understanding of what evaluation is and is not about.

Involving staff at every step: We value your organization’s resources, talent, and input. That’s why we want to meet with you at the beginning and throughout the evaluation process to understand your goals, to get your feedback about every stage of the project, and to answer any questions you might have. The ability to steer the evaluation always remains in your control, and your voice will always be heard.

Assistance with reporting: You may have specific and detailed plans for communicating the data gathered during the evaluation process. Or, you may be unsure how to make your data available to your clients or stakeholders in the most effective and useful ways. At ACET, we are able to help you determine what format(s) will best meet your needs and goals, and to create professional, influential reporting materials for web and print – whether your audience is your community, your board, or the federal government.

We know these elements help nonprofit organizations achieve success – whether that be in the form of renewed or increased funding streams, a clarified vision for future program expansion, or answers to burning questions – because we have been instrumental to the success of many nonprofits. You can find out more about these successes by viewing the Projects page of our website. If you are a nonprofit decision maker unsure of how evaluation fits into your work, let us help you to explore the possibilities.

ACET Staff

Three researchers from Florida’s Nova Southeastern University tackled the subject of bullying in an article recently published in the International Journal of Education Policy & Leadership. The report highlights the knowledge gap between teacher perceptions and understandings of bullying and what male and female students report.

A survey was conducted of 139 teachers and administrators from schools across the United States regarding bullying prevention training, curriculum, and meeting with the parents of bullies. The researchers found that most administrators and teachers agreed (at levels of 90% or above) that bullying prevention should be introduced into school curriculums at all levels and that more bullying prevention training for staff would be beneficial. These results came with an interesting gender nuance, in that female respondents were more likely to agree with the need for a bullying prevention curriculum than males, and males expressing more confidence in meeting with the parents of bullies and students affected by bullying. The survey also found that administrators were significantly more confident than teachers in meeting with the parents of students who bully and are bullied. The researchers stress that both these points warrant further study and the lack of total buy-in from teachers and administrators is a potential hindrance to the success of future bullying prevention efforts.

No matter what your position is on the matter, we’d like to hear from you about the subject of gender and bullying prevention in the comments.

Mary

Lopsided representation of men over women in the STEM (science, technology, engineering, mathematics) disciplines has been a subject of national study and initiatives for years, and one such study about why this exists came to our attention recently via a colleague.

“Problems in the pipeline”, published in the Journal of Applied Developmental Psychology in 2008, examines the effect of stereotype threat on girls in mathematic achievement. Stereotype threat is the disruptive effect of stereotypes on the performance of a group. Initially developed to explore the differences between the test scores of Caucasian and African-American students, the research team (Good, Aronson, and Harder) used stereotype threat in their work to examine the performance of women versus men via a college calculus test. Out of 174 participants, some were merely told by the instructor that the test was intended to determine why some individuals were better in math than others. Others were also specifically told that men and women scored equally well on the test. Data sources included calculus scores, course grades, and a mathematics self-efficacy survey.

The test results indicated that even at a high level, women answered fewer questions and scored lower when not told that men and women performed equally well. This occurred despite evidence that all tested women were as well if not more prepared for the test than men, and that their regular course grades were not significantly different compared to those of men. That is, the researchers concluded the only difference between women told that they did as well as men on the test and women who were not told this was the impact of the stress of stereotype threat. It is also of interest to note that both groups of women displayed the same lack of confidence in their answers no matter what they were informed of prior to taking the test. For the researchers, this further reinforced their belief that the weight of stereotypes was being brought to bear, leading to the obstruction of math as potential interests and careers for girls and women alike.

These results underline to us the critical role of the instructor in the classroom and the messages that they send to their students each and every day.

ACET Staff

ACETis very excited to share a study we recently completed for the Minnesota Department of Education. This study first developed a criterion for gauging the capacity and effectiveness of Minnesota Institutions of Higher Education (IHE) in preparing early childhood educators and care providers, then applied that criterion by examining Minnesota 2- and 4-year IHE early childhood programs serving nearly 3,000 students. ACET convened a 10-member advisory council, reviewed literature and available data, and conducted interviews with IHE staff to answer 2 questions:

1. What is the capacity of IHE to prepare early childhood educators and care providers?

2. What systems are in place at each IHE to ensure their effectiveness?

A report brief (opens PDF) was developed in June highlighting the key findings. If you are interested in viewing the more detailed results, the technical report and other background information can be found at www.acetinc.com/ECPP/AG.htm.

We are very excited about this release and hope you find the data informative. If you have any comments or questions about the brief, please share!

Joseph, Dan, and Stella

In a previous blog post, I introduced the topic of Executive Summaries as well as a brief look at the keys to an effective summary. In this post, I want to identify the key components of the structure and flow of a summary.

Heading: The heading, or title, of your Executive Summary should be brief but still descriptive. It should communicate its function to its readers immediately.

First Paragraph: Introduce your client organization, its source of funding, the name of the program, its goals, and its target population.

Second Paragraph: Describe the program and evaluation specifics. This is a good place to include historical information about the project if relevant. Explain how the program was evaluated, the number of individuals who participated, to what extent, and how those contributions were measured.

Third Paragraph: Use your third paragraph to address impact statements. Describe the key findings for each evaluation goal previously identified.

Fourth Paragraph: Describe the strengths and barriers, or successes and challenges, encountered in the evaluation. This is also a good place to explore any other notable or unintended outcomes experienced.

Final Paragraph: This paragraph will serve as your conclusion. Use this paragraph to make recommendations or suggest next steps.

Beyond content, design is also a factor in delivering information in a clear and uncluttered way. If your Executive Summary consists of pages and pages of paragraphs, it is likely exceeding the target length of 1 to 2 pages and/or has been cut and pasted in large pieces from its original report. Keep in mind that the purpose of the Executive Summary is to succinctly report the most important aspects of an evaluation report to stakeholders, not simply to produce an abridged version of the original report. Try using visually distinct headers to make the layout of the information clear (similar to the above bold words). Also, to only report the most important information, leave out charts, graphs, and tables in the construction of your Executive Summary; readers can find these components in the full report if they want further information.

Good luck.

Stella

A lot of work goes into producing a good executive summary of key evaluation report findings. Executive summaries are typically 1 to 2 pages in length, can be photocopied on 1 sheet of paper, and are easily distributed.

A good executive summary requires the evaluator to rethink the original report. This task should not be a cut-and-paste job from the body of the report. The evaluator literally has to step back, think beyond what a specific finding shows, and strategically tease out key information into a succinct summary.

Writing a good executive summary takes time and practice. The more that you do it, the better you get. Unless you are gifted at playing chess and can anticipate future moves of your client, don’t attempt to write the summary until the report is finished. Tackle this task last, or even consider sharing the report with clients and then facilitating a conversation with them to discuss key findings. This step in turn can enrich how you frame the summary.

Stay tuned for more about this subject in the future!

Stella

A new study about increasing classroom performance caught the eyes of staff at ACET and we wanted it to share it with you. The study linked regular recess and play opportunities to better classroom performance and a decrease in bullying and student conflict, and was conducted by Mathematica Policy Research and the John W. Gardner Center for Youth and Their Communities at Stanford University.

In the study, 14 schools across the United States partnered with Playworks, an Oakland, California non-profit devoted to providing low-income schools with physical activity options and other play events such as game times. Playworks provided schools with trained coaches and, in some cases, junior coach programs where students were recruited to monitor recess times alongside adults. Students, teachers, principals, and coaches were interviewed and surveyed about their experiences with the program and what changes, positive and negative, they saw in student relationships, academic outcomes, and classroom behaviors. Students and teachers reported feeling engaged by Playworks’ coaches’ programs and their use of positive messaging and conflict resolution strategies (for example, using a game of rock-paper-scissors to solve a dispute) as they pertained to playground interactions, though students did not report a decrease in student aggression or perception of aggression. Teachers with Playworks programs in their schools reported faster transition times between recess and learning than schools that did not have Playworks programs. Principals from all schools that received Playworks services reported a need for the organization’s presence in their schools for the coming year.

According to an advisor from the study’s sponsor, the Robert Wood Johnson Foundation, “These new findings… tell us that kids better relate with one another, resolve conflicts constructively, get plenty of physical activity on the playground, and return to class focused and ready to learn. Increasingly, health and education leaders are recognizing that recess and play are effective ways to strengthen schools and foster children’s social, emotional, and physical development.”

You can read more from the Robert Wood Johnson Foundation press release here or view the report here [opens as PDF].

Mary

You may be familiar with policy memos, those one-to-two page documents often delivered by policy analysts to inform decision makers. Policy memos are a remarkably useful tool for all sorts of information exchange. In many ways they are similar to executive summaries. Both policy memos and executive summaries:

• Effectively communicate key messages in a succinct manner;
• Present evidence from both sides of a debate (pros and cons or strengths and weaknesses); and
• Synthesize much information into one  easy to use document.

But there’s one key distinction between executive summaries and policy memos: Policy memos contain a call to action of some sort. Often, this takes the form of a recommendation to follow one course of action over another. Or, it may be a simple set of action steps.

At ACET we often use policy-type memos to communicate with clients, highlighting key findings, strengths, challenges, and next steps. You might want to consider whether this document style could be a similar useful tool in your organization.

Lately I’ve become preoccupied with a wonderful resource on visual information – vizthink. Vizthink’s focus is on effective visual communication in a variety of forms. The most obvious form of visual communication is presentations, but vizthink also has information and resources on sketching, infographics, facilitating conversations, videos, and interface designs. Vizthink offers articles/blogs, webinars, podcasts, and a list of resources for those who are passionate about finding new, interesting, or more effective ways to present data visually.

For instance, the link below shows a 2010 video by Alex Lundry, the VP and Director of Research at TargetPoint. This was part of a presentation to Ignite DC entitled “Chart Wars: The Political Power of Data Visualization.” In the video Alex talks about data visualization ‘sins’ that can be used to skew data so it is less accurate.

Chart Wars: The Political Power of Data Visualization

There are also vizthink groups on LinkedIn, Twitter, and Facebook through which you can have conversations with others who are interested in visual communication. I hope you find vizthink interesting and provocative!

Kirsten

The 2001 No Child Left Behind (NCLB) law set a new standard of proficiency rates for all students across the United States. The law specifies that proficiency proportions for mathematics and reading are required to be 100% by 2014 across all 50 states and territories.

The NCLB law also stipulated that each state could determine which test would be used to measure students’ proficiency. And the NCLB law ensured states that they would determine academic standards, minimum proficiency for graduation, and how to test for student proficiency. Because states can set their own academic standards and how to test for student proficiency, one would expect the difficulty of obtaining proficiency to vary from state-to-state. In fact, under NCLB it is possible that 100% of a group of students might be proficient in State A, but only 70% of the same group of students might be proficient in State B because State B has higher academic standards or a more difficult proficiency test.

So how can we identify differences in states’ academic standards if each state sets their own standards and has their own proficiency test? One way is to test students in different states on the same test and compare the results to proficiency results. One such test is the Measures of Academic Progress (MAP) which is published by the Northwest Evaluation Association. Many states use the MAP to measure students’ progress towards proficiency. Staff at Northwest Evaluation Association’s Kingsbury Center examined potential differences in proficiency measures for 36 states. Kingsbury Center staff identified the MAP percentile scores that were needed to pass each state’s test and compared the differences in difficulty on mathematics and reaching proficiency across states. The higher the percentile needed to pass the test, the more difficult the test.

So where does Minnesota stand? As can be seen in the table below, with all grades aggregated into a single state average, Minnesota’s mathematics standards were the 4^th most difficult of the 36 states studied. Students in Minnesota had to score just above the 45^th percentile on the mathematics MAP in order to be proficient, but the average score for all 36 states was around the 35^th percentile to be proficient in mathematics. Across states, mathematics proficiency varied – some students had to score at the 65^th percentile to be proficient while students in another state had to score at the 10^th percentile. In reading, Minnesota’s reading standards were the 14^th most difficult out of the 36 states studied. In order to be proficient in reading, students in Minnesota had to score at the 35^th percentile, but the average score to be proficient among the 36 states was at the 30^th percentile. Again, across states there was a wide range of proficiency for reading – some students had to score at the 50^th percentile to be proficient while other students had to score only at the 8^th percentile.

*The higher the percentile, the more difficult it is to for a student to score proficient on a state test.

The current Obama administration recognizes the challenges that the existing NCLB law presents for establishing proficiency. The Obama administration’s blueprint for reauthorizing the NCLB Act emphasizes college and career readiness rather than proficiency standards. Perhaps, moving forward with new college readiness standards, the reauthorization might yield some means of standardizing results between states so we can remove some of the big gaps between standards.

To learn more, please click this link for the recent MAP report from the Kingsbury Center: http://kingsburycenter.org/sites/default/files/State%20of%20Proficiency%202011_0725.pdf

Joseph