ACET, Inc. Blog

The Blue Cross and Blue Shield of Minnesota Foundation has announced the “Connect for Health Challenge.” The goal of Connect for Health is to strengthen social connectedness in low-income communities across Minnesota. Up to $500,000 will be awarded to nonprofits, schools, and local units of government using InCommons, a community-based online initiative that connects people-online and face-to-face-so that they can find, use, and share knowledge and resources while addressing social problems.

Eligible agencies must be based in Minnesota and serve Minnesotans. Up to 20 grants of $20,000 each will be awarded and one agency will receive up to $100,000. All agencies that submit a grant application by April 27, 2012 will be eligible to win a free two-minute video of their organization shot and edited by a professional videographer.

For more information about the Connect for Health program and how to enter, please visit: http://www.incommons.org/ConnectforHealth

Good luck!

Kirsten

We are pleased to announce that ACET has an opening for a new Evaluation Specialist! The newly hired Evaluation Specialist will be a highly motivated and enthusiastic individual with demonstrated experience performing evaluation or research studies and writing clear, concise reports for a variety of audiences. Experience with relational databases in Excel and SPSS (or other statistical software) is required.

A qualified candidate will have a master’s degree in the field of evaluation, health science research, policy analysis, or related field. A degree of creativity, independence, and latitude is expected with this position. Job duties for this position include:

1. Working in a team setting to plan an evaluation (e.g., logic model development, evaluation planning, database mapping);

2. Conducting literature and document reviews;

3. Designing qualitative and quantitative measures (e.g., focus group questions, survey items, interview questions, rubrics, site visit protocols);

4. Collecting data and co-facilitating meetings;

5. Compiling results, preparing reports, and documenting progress toward outcomes;

6. Working with clients to build evaluation capacity;

7. Sharing ideas on program evaluation methodologies and frameworks; and

8. Assisting with various aspects of business operations and other relevant duties as assigned.

Applicants should email or send a one-page letter of interest and a resume (not to exceed two pages) to:

ACET, Inc.

9868 Lyndale Avenue South

Minneapolis, MN 55420

Email: info@acetinc.com

To learn more about ACET and the kind of work we do, please visit: http://minneapolis.craigslist.org/hnp/hum/2918713510.html.

I’ve been thinking a little bit about the house building process recently – not thinking of undertaking it, but thinking about how embarking on an evaluation can be surprisingly similar to deciding to build a house. There’s no doubt that designing and building a new home is a gargantuan task and the experience can feel completely overwhelming. But, for those who put the effort in and survive to tell the tale, the results can be superbly rewarding.

So what makes me think that house building is so similar to building an evaluation? For one thing, the first step is basically the same: You call in the experts. For home building, you find a builder and for an evaluation, you find an evaluator. But what really got me thinking about the similarities is this: in either case, once you’ve called in the expert, you have many options of how you want to proceed. In my mind, people fall into three general approaches on either of these projects:

1) “I want to be involved, but I need help figuring things out.”

For this person, the expert becomes a facilitator, who helps them identify their needs and options. For instance, a builder might ask a client questions about how often and what kind  of cooking she/he does, in order to figure out how to layout the kitchen. Similarly, an evaluator might ask a client to list their current needs (e.g., areas of program strengths and challenges) and options (e.g., capacity of the program staff) in order to determine an appropriate evaluation plan.

2) “I’m happy to let the experts handle things.”

This is the person who hires a builder because she/he likes their work and says, “Design me a home.”  This person knows they’ll be happy with the results because they trust the expert, and they don’t have the time or the desire to get involved in the details.  Many organizations hire an evaluator with exactly the same idea of trusting the expert to handle the planning and execution of the entire evaluation process.

3) “I have a really good idea of what I want.”

This person knows exactly how many doors they want in their house—and what they want the doorknobs to look like as well. When working with an evaluator, this type of client has clear expectations about how they want the process to go. For this client the evaluator is contracted to perform specific tasks such as data analysis or on an as-needed basis.

My list above of course is not meant to be exhaustive or inclusive of all scenarios. Given the complexity of evaluations and environments, it is likely that there is a good mix of all three types in a single client. If you have other ways to work with an evaluator, please share!

Stella

A very interesting article came across my desk a couple of weeks ago. In it, the authors describe their investigation of wiki use in U.S. K-12 schools. As you may know, a ‘wiki’ is a website to which users can contribute knowledge and information. Wiki’s are edited in real-time through a web browser. Two popular wikis are Wikipedia (an online encyclopedia) and Wiktionary (an online dictionary). Wikis, in general, are hosted at specific websites and are often free. Anyone can start their own wiki on any topic or can contribute to others’ wikis.

Reich, Murnane & Willet were specifically interested in how wikis were being used in schools and whether or not wikis were being used to promote the development of students’ digital literacy. Reich et al. sampled 1% of the wikis from a site popular for hosting free, educational wikis and 255 of those sites could be linked to K-12 schools in the U.S. Each wiki was assessed to determine how the site was being used. Specifically, the assessment tool included the following questions, and each was answered “Yes” or “No”:

• Do students use the wiki to access classroom materials?
• Are students the primary contributors to the wiki?
• Do students credit the sources of their contributions to the wiki, such as a citation (for books or paper material) or a hyperlink for digital information?
• Do students use text formatting, such as bold or italicized text or bulleted lists?
• Do students respond to each other’s contributions to the wiki?

Each wiki was assessed on 24 unique characteristics and total scores were computed.

Although Reich et al. examined the general characteristics of the wikis, I found the most interesting section of the article to be an examination of wikis used in schools serving larger proportions of lower-income families (Title I-eligible schools) and other schools (non-Title I eligible). Reich et al. found substantial differences in wikis developed at Title I and non-Title I eligible schools:

• More wikis in Title I-eligible schools remained undeveloped (e.g., contained an auto-generated front page only) or were teacher-only sites (50% for Title I-eligible schools and 30% for non-Title I);
• There were also differences in the median time the wiki remained active for Title I and non-Title I eligible schools (6.5 days for Title I-eligible and 32 days for non-Title I eligible schools); and
• Fewer wikis at Title I-eligible schools promoted high levels of student digital literacy compared to non-Title I eligible schools (17% at Title I-eligible and 36% at non-Title I eligible schools).

In other words, although the “space” for educational wikis is free to all teachers and their students, not all students are afforded the opportunity to develop high-level skills in digital literacy that could be learned in a wiki environment. Fewer students in schools with larger proportions of lower-income families have the opportunity to improve their digital literacy through wikis while more students in other communities can develop their digital literacy skills. The authors concluded the article with a description of how students’ online contributions to wikis also have the potential for student assessment.  As Reich et al. point out, online contributions to wikis by students represent long-term learning of how to solve nebulous, ill-structured problems and how to collaborate with others. In addition, using data from students’ online contributions to assess learning is incredibly efficient compared to traditional assessments because online contributions are integrated into the learning process. As a result, there is no loss of instructional time in order to assess what students have learned!

For more information, please see:

Reich, J., Murnane, R., & Willett, J. (2012). The state of Wiki usage in U.S. K-12 schools: Leveraging web 2.0 data warehouses to assess quality and equity in online learning environments. Educational Researcher, 41, 7-15. doi: 10.3102/0013189X11427083

Kirsten

Have you ever been asked to attend a meeting and you were unclear about the meeting purpose? What about a meeting or conference call with more people than the subject or time could accommodate? How about a meeting that was called at the last minute and was not well planned? Or one in which the technology would not function?

Often, it’s easy to spot what makes a ‘bad’ meeting, but what makes a ‘good’ meeting is often more difficult to determine. A substantial part of a good meeting occurs behind the scenes with the facilitator’s pre-meeting preparation. When facilitators’ take the time to carefully prepare, the meeting is usually smoother, more productive, and less stressful for everyone involved. So what should a facilitator focus on during preparation? In my experience, and review of articles on conducting successful meetings, I have found that to conduct the most effective meeting the facilitator should:

• Have a clear objective for the meeting. Meet for a purpose, not just “because you can” or “think you should.”
• Have an agenda for the meeting. Know what goals or objectives you would like to achieve and share that information ahead of time with the meeting participants.
• Be mindful of the number of participants you invite when feedback is required. With too many people, it is difficult for all participants to share their thoughts and for decisions to be made. Although many contextual factors (amount of necessary feedback, interpersonal relationships, time allocation, etc.) will contribute to how many people you should invite, some of the literature has suggested that no more than 12 people make for a good environment for everyone to be heard.
• Send electronic copies of meeting documents to members beforehand. This will allow all participants to adequately prepare themselves for the meeting and to contribute to achieving the meeting’s goals.
• Give everyone enough time to prepare for the meeting. For a meeting that lasts at least one-hour, anticipate 3-5 business days for preparation.
• Ensure technology is working appropriately. Test the technology prior to the meeting to make certain it will work the way you want.

If you are interested in successful meeting facilitation and would like to learn more, try these websites:

• http://www.arma.org/bookstore/files/Francisco.pdf
• http://www.mindtools.com/CommSkll/RunningMeetings.htm
• http://www.dummies.com/how-to/content/conducting-effective-business-meetings.html

Are there any keys or strategies you have found particularly effective? If so, please share!

Dan

Non-Equivalent Dependent Variables (NEDV) may sound complicated but, like a number of other techniques in statistics, the idea is actually pretty straightforward. In essence, NEDV are two (or more) unequal variables that are used together to detect change. Let me illustrate with an example. Say we have a great new method to teach grade school kids how to spell a certain number of words; let’s say 10. We are using a new way of spelling that we think will have a great effect on kids’ ability to spell, so we test kids on the 10 words before we teach them with our new technique (pretest) and then again a few days after we teach them (posttest). We expect to find that kids will get very few if any words right on the first test and most or all right on the second test and this will show that our teaching method was effective. However, you might worry that improvements from pre- to posttest might be due to something other than the instruction we are providing. For example, many children’s television shows teach spelling or parents may work with their kids at home on spelling. If kids received this extra support above and beyond your instruction and showed great results on your 10 word spelling posttest we could not be sure how much of their posttest performance was due to our program or caused by the extra support. Maybe without our program kids would have done as well with just the extra support and our program is wasting resources.

NEDV are used in situations like the one just described because they can be effective at detecting if external factors impacted the results of a study (like our pretest/posttest example). In a typical pretest/posttest study, kids would be asked to spell 10 words on the pretest and spell the exact same 10 words on the posttest; this would be ‘equivalent dependent variables’ because the words are identical. But in an NEDV study kids would spell 10 words on the pretest and 20 words on the posttest: the 10 words from the pretest and 10 new words. We would expect that if kids are receiving extra help they would correctly spell some of the 10 new words we did not teach them, and if we see that kids indeed correctly spelled those 10 new words then we can start to doubt the observed effects of our program on the 10 words we taught them. But, if we see that kids do not correctly spell the 10 new words but do correctly spell the 10 words we taught, then we can be more confident in the observed effects.

Joseph