ACET, Inc. Blog

Being an effective writer is an important skill for an evaluator. It takes time and practice. One of my favorite books on effective writing was written by Morris, Fitz-Gibbon, and Freeman, titled “How to Communicate Evaluation Findings.” Only 92 pages long, this book is indispensable at helping evaluators to communicate findings in a succinct way and is full of creative examples and advice. Here’s one of my favorite parts, regarding the choice of the right form for communicating findings to a given audience (page 31):

You are eager, perhaps, to show the audience the clarity and brilliant complexity of the evaluation plan, the quick-thinking administrative maneuvers that saved the evaluation from disaster, or the mathematical genius so obviously guiding the analyses. DO NOT DO IT! You will bore the audience to death and render the findings indigestible.

This excerpt sums up for me the beauty and the utility of this book, understanding those things that evaluators value in reports, and reminding us that they aren’t the same things that stakeholders and other audience groups value.

Stella

Lopsided representation of men over women in the STEM (science, technology, engineering, mathematics) disciplines has been a subject of national study and initiatives for years, and one such study about why this exists came to our attention recently via a colleague.

“Problems in the pipeline”, published in the Journal of Applied Developmental Psychology in 2008, examines the effect of stereotype threat on girls in mathematic achievement. Stereotype threat is the disruptive effect of stereotypes on the performance of a group. Initially developed to explore the differences between the test scores of Caucasian and African-American students, the research team (Good, Aronson, and Harder) used stereotype threat in their work to examine the performance of women versus men via a college calculus test. Out of 174 participants, some were merely told by the instructor that the test was intended to determine why some individuals were better in math than others. Others were also specifically told that men and women scored equally well on the test. Data sources included calculus scores, course grades, and a mathematics self-efficacy survey.

The test results indicated that even at a high level, women answered fewer questions and scored lower when not told that men and women performed equally well. This occurred despite evidence that all tested women were as well if not more prepared for the test than men, and that their regular course grades were not significantly different compared to those of men. That is, the researchers concluded the only difference between women told that they did as well as men on the test and women who were not told this was the impact of the stress of stereotype threat. It is also of interest to note that both groups of women displayed the same lack of confidence in their answers no matter what they were informed of prior to taking the test. For the researchers, this further reinforced their belief that the weight of stereotypes was being brought to bear, leading to the obstruction of math as potential interests and careers for girls and women alike.

These results underline to us the critical role of the instructor in the classroom and the messages that they send to their students each and every day.

ACET Staff

Lately I’ve been very interested in how surveys may be better used and administered. One hurdle with almost every survey is the number of people who actually complete the survey (compared to those who are given the survey known as a “response rate”). Recently I came across an article that described the use of “norm-based” appeals to increase survey response rates (Misra, Stokols & Heberger Marino, 2012). In this context “norm-based” refers to societal norms such as the direction you face when you get in an elevator or the distance you maintain between yourself and a stranger.

Misra et al. studied the effectiveness of using normed-based appeals to increase survey response rates. At a conference to promote interdisciplinary research all participants were asked to complete a survey immediately after the conference ended and then again three months later. Half of the participants received a generic message inviting them to complete the survey and half received a norm-based appeal in which the behavior of other people was described. The norm-based appeal given to half of the participants read, “Most years, over 75% of conference participants complete the survey. Please join your fellow participants in improving the quality of future conferences by filling out the survey.” At the three month follow-up participants were invited to complete another survey and half received the generic message and half received the norm-based appeal. In all, there were four groups in the study:

Misra et al. found that participants who received the norm-based appeal immediately after the conference (Groups 3 and 4) were more likely to complete the survey than those who received the generic message (Groups 1 and 2). Also, participants who received the norm-based appeal at both time points (Group 4) were more likely to complete the second online survey than those who received generic messages at both time points (Group 1). And participants who received both the generic message and the norm-based appeal (Groups 2 and 3) fell somewhere in between Groups 1 and 4. In summary, the results here suggest that norm-based appeals may be an effective technique to improve survey response rates.

For more information, please see:

Misra, S., Stokols, D., & Heberger Marino, A. (2012). Using norm–based appeals to increase response rates in evaluation research: A field experiment. American Journal of Evaluation, 33, 88-98. doi: 10.1177/1098214011414862

Do you have any methods you use to increase response rates with your online surveys? If so, please share!

Dan

A very interesting article came across my desk a couple of weeks ago. In it, the authors describe their investigation of wiki use in U.S. K-12 schools. As you may know, a ‘wiki’ is a website to which users can contribute knowledge and information. Wiki’s are edited in real-time through a web browser. Two popular wikis are Wikipedia (an online encyclopedia) and Wiktionary (an online dictionary). Wikis, in general, are hosted at specific websites and are often free. Anyone can start their own wiki on any topic or can contribute to others’ wikis.

Reich, Murnane & Willet were specifically interested in how wikis were being used in schools and whether or not wikis were being used to promote the development of students’ digital literacy. Reich et al. sampled 1% of the wikis from a site popular for hosting free, educational wikis and 255 of those sites could be linked to K-12 schools in the U.S. Each wiki was assessed to determine how the site was being used. Specifically, the assessment tool included the following questions, and each was answered “Yes” or “No”:

• Do students use the wiki to access classroom materials?
• Are students the primary contributors to the wiki?
• Do students credit the sources of their contributions to the wiki, such as a citation (for books or paper material) or a hyperlink for digital information?
• Do students use text formatting, such as bold or italicized text or bulleted lists?
• Do students respond to each other’s contributions to the wiki?

Each wiki was assessed on 24 unique characteristics and total scores were computed.

Although Reich et al. examined the general characteristics of the wikis, I found the most interesting section of the article to be an examination of wikis used in schools serving larger proportions of lower-income families (Title I-eligible schools) and other schools (non-Title I eligible). Reich et al. found substantial differences in wikis developed at Title I and non-Title I eligible schools:

• More wikis in Title I-eligible schools remained undeveloped (e.g., contained an auto-generated front page only) or were teacher-only sites (50% for Title I-eligible schools and 30% for non-Title I);
• There were also differences in the median time the wiki remained active for Title I and non-Title I eligible schools (6.5 days for Title I-eligible and 32 days for non-Title I eligible schools); and
• Fewer wikis at Title I-eligible schools promoted high levels of student digital literacy compared to non-Title I eligible schools (17% at Title I-eligible and 36% at non-Title I eligible schools).

In other words, although the “space” for educational wikis is free to all teachers and their students, not all students are afforded the opportunity to develop high-level skills in digital literacy that could be learned in a wiki environment. Fewer students in schools with larger proportions of lower-income families have the opportunity to improve their digital literacy through wikis while more students in other communities can develop their digital literacy skills. The authors concluded the article with a description of how students’ online contributions to wikis also have the potential for student assessment.  As Reich et al. point out, online contributions to wikis by students represent long-term learning of how to solve nebulous, ill-structured problems and how to collaborate with others. In addition, using data from students’ online contributions to assess learning is incredibly efficient compared to traditional assessments because online contributions are integrated into the learning process. As a result, there is no loss of instructional time in order to assess what students have learned!

For more information, please see:

Reich, J., Murnane, R., & Willett, J. (2012). The state of Wiki usage in U.S. K-12 schools: Leveraging web 2.0 data warehouses to assess quality and equity in online learning environments. Educational Researcher, 41, 7-15. doi: 10.3102/0013189X11427083

Kirsten

Although we focus on research and evaluation tips or grant leads on this blog, we also understand the need for some lighthearted posts – that’s why we occasionally have staff book reviews. Many of you may know that I majored in English in college, which would lead you to the obvious assumption that I like to read. You may not know, however, that one of my all-time favorite authors is Dr. Seuss. My parents introduced me to Dr. Seuss when I was a very young child – I even taught myself to read using Dr. Seuss’ books when I was 3! His works stuck with me as I grew up. To this day, my favorite Dr. Seuss book is “The Sneetches” and I would give anything to own a one-piece Sneetch pajamas!

You can imagine my interest when I discovered “The Seuss, the Whole Seuss, and Nothing But the Seuss,” by Charles D. Cohen. By reading the book, I learned how Theodor Seuss Geisel gradually adapted his pen name to Dr. Seuss after he began publishing in “Judge” magazine (he was adding mock authority to a cartoon he drafted). I learned that Dr. Seuss had many varying explanations for his odd characters, so one rarely knew the accurate story of how he created them. Dr. Seuss was well-known among his schoolmates and peers as a clever and witty person – in fact, he was penning whole pieces of parody when he was only 14 years old. Although I knew that Dr. Seuss had done far more than authoring children’s books, I found that he also created adult books, publications, illustrations, paintings, sculptures, screenplays, and more! And, of course, I learned the history of “The Sneetches.” Dr. Seuss created the characters to promote equality in a world of intolerance after seeing multiple court cases involving racial segregation.

Cohen’s detailed biography of Dr. Seuss proved to be an interesting, and informative read for any Seuss bibliophile curious about the life and ideas of the popular author. The book is also filled with pictures and illustrations – some of which were nearly lost and required much research and dedication on the behalf of Cohen. Cohen’s book is particularly useful because much of the information out there is cursory or even incorrect. I appreciate that I now have a deeper understanding of a man who has influenced my entire life through his works.

Cassie

Tears in the Darkness: The Story of the Bataan Death March and Its Aftermath
by Michael Norman and Elizabeth M. Norman

Both of my grandfathers served with the American armed forces during World War II: one was a sergeant for the Army in the Pacific and the other was an Army Air Corps co-pilot of a bomber based out of England. Although neither of my grandfathers spoke of their experiences during that time, lately I’ve found myself drawn to a wide collection of works regarding World War II, including my latest read, Tears in the Darkness: The Story of the Bataan Death March and Its Aftermath by Michael and Elizabeth M. Norman.

Although the title suggests the book is primarily about the Bataan Death March, I found it to be slightly misleading. In reality, the book follows the experiences of Ben Steele, a Battle of Bataan survivor, from his arrival in the Philippines in late fall of 1941, the battle for the Bataan peninsula, the ‘death march’ to the POW camps, transportation to Japan to work in a mine, and Steele’s return to the United States in 1945. Several stories of others who fought on Bataan, including members of the Japanese Imperial Army, are also interwoven with Steele’s story.

Tears in the Darkness is not a book for everyone. Parts of the story, especially descriptions of how POWs were treated, both soldiers and civilians, are difficult to read. But the Normans also include descriptions of what happened in Japan’s politics and milieu of the army that allowed war crimes in the Pacific during WWII to occur. As a result, I feel I have a much better understanding of what cultural events and context led to the horrors in the Pacific. (Sadly, I see many of those cultural events being repeated today in other parts of the world.) The authors also describe Ben’s life after returning to the U.S. and, as a result, I think I better understand why my grandfathers never shared their experiences with their grandchildren.

This July I participated in a “multi-tier” review of grant applications for the U.S. Department of Education. It was a great experience! I learned a lot about how grants are reviewed at the Federal level and would like to share those processes and some advice with you.

Review Processes

Under a multi-tier review, the required proposal elements are divided into sections and reviewers assess and score specific sections of the proposal. Reviewers are assigned to different tiers based on their level of expertise. At the end of each tier of reviewing, the highest scoring applications move on to the next tier for additional review. Checks-and-balances are built into the entire process to ensure that the entire review process is fair and equitable.

For example, I was a Tier II reviewer and the applications had already been scored on four proposal elements. The Tier II reviewers read and scored the applications for two additional sections: the research foundation of the application and proposed evaluation activities (we did have to read the entire proposal, but scored only those two sections). After reading the assigned applications, I participated in a conference call with another reviewer assigned to the same applications and a panel monitor (an employee of the U.S. Department of Education). During the conference calls, the reviewers discussed the merits and challenges of each application and how well each met the scoring criteria. The reviewers also worked towards aligning the assessments so that we were consistent in our application of the review criteria for all of the applications.

Grant Application Advice

Three key themes emerged from my review:

1 – Make sure the application covers all of the required elements outlined in the RFP.
One of the proposals I read did not include background research or theory in the narrative. As a result, the reviewers scored that section quite low. Although this advice may feel self-evident, it’s a pretty good bet that elements listed as ‘required’ in the RFP will be included as scorable criteria in the review phase. It is very important and very beneficial to have someone read the application and verify that it covers all required elements before submitting it.

2 – Ensure space and point allocations are proportional to application pages.
One of the applications I read included more than five pages describing background research (worth up to 10 points) and but fewer than five pages describing the proposed evaluation (worth up to 15 points). While the research section was very detailed (and was scored highly), the description of the proposed evaluation was very short and the lack of detail resulted in a lower score for the proposed evaluation section. When writing a grant application, it’s important that sections that are worth more points be allocated more space in the proposal than sections worth fewer points. This is especially true when there is a page limit to the proposal!

3 – Review, re-read, and edit the application to ensure an appropriate breadth and depth of the narrative.
Several of the applications I scored included descriptions of proposed evaluation activities that were so broad and sweeping it was difficult to determine what the evaluation would be. With highly competitive grant competitions such as the one I reviewed for, it’s very important to give the reviewers enough breadth and depth in the narrative so they get a sense of proposed program activities, why the program is needed, when the program has been successful in the past, and how the proposed program will be evaluated.

I hope this information is useful to you and your agency when you prepare an application for a grant. Please comment below or email me if you have any questions or would like additional information!

Kirsten