ACET, Inc. Blog

Lopsided representation of men over women in the STEM (science, technology, engineering, mathematics) disciplines has been a subject of national study and initiatives for years, and one such study about why this exists came to our attention recently via a colleague.

“Problems in the pipeline”, published in the Journal of Applied Developmental Psychology in 2008, examines the effect of stereotype threat on girls in mathematic achievement. Stereotype threat is the disruptive effect of stereotypes on the performance of a group. Initially developed to explore the differences between the test scores of Caucasian and African-American students, the research team (Good, Aronson, and Harder) used stereotype threat in their work to examine the performance of women versus men via a college calculus test. Out of 174 participants, some were merely told by the instructor that the test was intended to determine why some individuals were better in math than others. Others were also specifically told that men and women scored equally well on the test. Data sources included calculus scores, course grades, and a mathematics self-efficacy survey.

The test results indicated that even at a high level, women answered fewer questions and scored lower when not told that men and women performed equally well. This occurred despite evidence that all tested women were as well if not more prepared for the test than men, and that their regular course grades were not significantly different compared to those of men. That is, the researchers concluded the only difference between women told that they did as well as men on the test and women who were not told this was the impact of the stress of stereotype threat. It is also of interest to note that both groups of women displayed the same lack of confidence in their answers no matter what they were informed of prior to taking the test. For the researchers, this further reinforced their belief that the weight of stereotypes was being brought to bear, leading to the obstruction of math as potential interests and careers for girls and women alike.

These results underline to us the critical role of the instructor in the classroom and the messages that they send to their students each and every day.

ACET Staff

Imagine that you’ve hired an evaluator to distribute a survey and report the findings back to you. You and your evaluator work hard to develop key questions for your constituents and distribute the survey in such a way to maximize the response rate. Your evaluator has crunched all the numbers and written a report, but you still have some questions. You wonder if looking at some individual survey responses might shed more light on your program and improvements which could be made, or you want to examine the responses of clients who fit specific demographic characteristics (e.g., how did women in outstate-Minnesota respond vs. men in the Twin Cities metro?).

At a glance, those follow-up questions may not seem problematic but from an evaluator’s perspective they can be ethically challenging. Evaluators have a responsibility to protect the anonymity of participants’ responses, especially if anonymity was promised at any stage of the survey process. In surveys, anonymity often comes from combining multiple responses together so that no individual’s response can be identified. In general, the more responses that are combined together the more anonymity participants have. So when clients want to examine an individual’s response or examine the responses of a small number of individuals, anonymity can be jeopardized.

There really are no “hard and fast” rules on how the minimum number of individuals needed to report disaggregated outcomes, but many agencies use 10 as the break point. In other words, if disaggregating outcomes results in 10 or more people in each group, the agency will report on the data. But if there are 9 or fewer people in each group, no data would be reported for that group. But each agency—and evaluator—has their own break point for reporting data; some use 10, others may go as low as 5, while others will not disaggregate data over concerns that any disaggregation could cause harm.

While planning an evaluation it is important for clients and evaluators to discuss all phases of the evaluation, including reporting. Clients can facilitate the reporting process by explaining their needs, including any needs or wishes for disaggregation. Doing so will give the evaluator time to adequately plan for collecting enough data so that the clients’ disaggregation needs are addressed. And evaluators should share their guidelines for break points so that clients are aware of any reporting limitations they might experience.

If you have questions about reporting, especially reporting small numbers of clients, please feel free to comment below and we’ll get back to you!

Stella

The concept of race and the associated terminology has altered over time. Dr. Michael Rodriguez, associate professor of Educational Psychology at the University of Minnesota, sent me a very intriguing example of the evolution of the race item on the U.S. Census. The census is administered every 10 years, and almost every administration seems to contain a variation in the language surrounding race. The full pictorial history of its evolution can be found here: http://racebox.org/

Some interesting things of note:

1. “Colored persons” started to be counted in 1830, and then only if they were classified as “free” (although the three-fifths compromise, allowing African slaves to count as three-fifths of a person for representation and distribution of taxes, began in 1787).
2. The distinction of counting only “free” people disappears after 1850 (the Civil War began in 1861).
3. The item specifically concerning “…of Spanish/Hispanic origin or descent” begins in 1980.
4. Also beginning in 1980, “color” (the color of a person’s skin) is no longer affiliated with race in the item stem.

Anything interesting you see? Leave us a comment and let us know.

Joseph

We often get requests to provide feedback on surveys. As we review the surveys, we tend to see three challenges occur quite frequently. I’ve written this blog to offer suggestions for addressing each of the challenges.

1. Double-Barreled Questions. In a double-barreled question, two concepts or ideas are included in one survey question. For example, the three survey items below are all double-barreled questions:
○        “How much of your monthly income is spent on restaurant dining and entertainment?”
○        “How often do you volunteer your time or make charitable donations?”
○        “How often do you consume beer and smoke cigars?”

But how does a respondent answer a double-barreled question? For example, if a family spent no money on restaurant dining but 10% of their income on entertainment, how should they respond? 0%? 10%? Average the two and respond 5%? Usually survey respondents do provide an answer, but they often utilize an individualized strategy for finding a response, which can be challenging for program staff to interpret.

Double-barreled questions can be addressed by limiting each survey item to one – and only one – concept. Below is a double-barreled question followed by revisions that eliminate the double-barrel:

2. Imbalanced Response Options. Imbalanced response options refer to a set of response choices that do not cover the range of possible choices. An imbalanced response scale is problematic because it limits respondents’ responses to only one end of a scale and may produce inaccurate results. For example, if a respondent wants to answer “strongly disagree” to a question but that option is not available, the survey is not capturing the respondent’s perspective. Using a set of balanced response options ensures that respondents’ answers can be accurately captured. A very good resource for a variety of balanced response scales can be found here: http://www.gifted.uconn.edu/siegle/research/Instrument%20Reliability%20and%20Validity/Likert.html

Below are some examples of imbalanced response options with improvements to the scale:

3. Overlapping Response Options. Overlapping response options simply means that there is some overlap in the response choices available to the respondent. For example, the following survey item has overlapping response options:

How many books did you read in the last month?
0 or 1 book
1 or 2 books
2 or 3 books
3 or more books

Overlapping response options are problematic because they can confuse the survey respondent. If you read two books in the last month, which option would you select: “1 or 2 books” or “2 or 3 books”? In addition, questions with overlapping response options are a challenge to interpret. How do you know how many books were read if the response options overlap? Overlapping response options can be addressed very easily by carefully constructing the response option list. Simply revise and edit the list so that there is no overlap between the response choices.

If you have any questions about the above challenges to designing a survey, please feel free to leave a comment or email me at Kirsten@acetinc.com. Please also contact any of us at ACET if you have questions about survey design in general.

Kirsten

ClearWay Minnesota℠ and the Minnesota Department of Health recently released results of the 2010 Minnesota Adult Tobacco Survey (MATS). Overall, news about tobacco use appears positive: adult smoking rates have continued to decline from 22.1% in 1999 to 16.1% as reported in the latest survey; exposure to secondhand smoke has decreased from 56.6% to 45.6% since 2007; and 87.2% of homes have gone smoke-free – up from 83.2% in 2007.

The results of the survey also pointed to areas of concern, such as a drastic increase in the use of smokeless tobacco products in both cigarette smokers and non-cigarette smokers. In fact, smokeless tobacco use for cigarette smokers has more than doubled since 2007, rising from 4.4% to 9.6%.

To view the full report and a fact sheet, click here: http://www.mnadulttobaccosurvey.org/

Cassie

Every three years, several Minnesota state departments (Education, Employment and Economic Development, Health, Human Services, and Public Safety) collaborate to administer the Minnesota Student Survey. The survey was last administered in 2010 to primary and secondary students (grades 6, 9, and 12) to assess their attitudes, beliefs, perceptions, and behaviors on a variety of topics. The survey covers areas such as students’ background, school, activities, heath, and behavior. A total of 335 school districts participated in the survey with a total of 130,908 students taking the survey across the state.

Results are reported by grade and the corresponding trends between grades are very similar; students appear to be changing attitudes, perceptions, beliefs, and behaviors nearly simultaneously. Many positive changes can be seen between recent years. For example, when comparing twelfth graders’ answers on the 2010 administration with the 2007 responses, several positive differences can be seen:

• More twelfth graders reported they were planning to attend college (88.9% in 2010 compared to 85.8% in 2007)
• More twelfth graders reported being physically active on five or more days in the past week (43.4% compared to 41.4%)
• Less twelfth graders reported alcohol usage in the past year (55.3% compared to 62.5%)
• Less twelfth graders reported binge drinking (five or more drinks in a row in the past two weeks; 23.4% compared to 29.0%)
• Less twelfth graders reported driving a motor vehicle after drug or alcohol usage (18.0% compared to 23.9%)
• Less twelfth graders reported smoking in the past 30 days (19.2% compared to 22.8%)

For more information follow the links below.

Select trends: http://education.state.mn.us/mdeprod/idcplg?IdcService=GET_FILE&dDocName=019017&RevisionSelectionMethod=latestReleased&Rendition=primary

2010 complete data tables: http://education.state.mn.us/mdeprod/idcplg?IdcService=GET_FILE&dDocName=019009&RevisionSelectionMethod=latestReleased&Rendition=primary

Joseph

Last August Heather and Kirsten offered a free workshop on survey design, which covered how to write useful items for your survey and common pitfalls to avoid. The feedback on the workshop was very positive and all attendees seemed to benefit from the material. ACET also offered a complementary review of an attendees’ existing survey as a means of following-up for the event.

We are pleased to announce that our second workshop has been scheduled! The Results are In: Analyzing and Reporting Survey Data for Stakeholders will cover common options for analysis of your survey data, guide you through the analysis process, and offer you suggestions for sharing your information with stakeholders in a way that is useful and meaningful.

The Results are In: Analyzing and Reporting Survey Data for Stakeholders will be held on:

Tuesday, February 23rd, 2010 from 1:00 to 3:00 pm at the
Neighborhood House Wellstone Center
179 Robie Street East, Saint Paul, MN 55107
Wellstone Center phone: (651) 789-2542

Seats are limited to this FREE event so please RSVP to Heather Scholz via email at heather@acetinc.com or via phone at the number below by Thursday, February 18th, 2010. For more information about ACET please visit our website at www.acetinc.com or contact our office at 952-922-1811.