ACET, Inc. Blog

When ACET staff met to debrief about the recent AEA Conference in Minneapolis, they had lots of interesting and useful information that they gleaned from days of attending presentations and discussions to share. Here are some comments.

Kirsten: I gravitated towards presentations on data visualization and reporting and sessions in which a skill or technique was demonstrated. In one of the sessions, Simon Geletta (Des Moines University) showed how to build information dashboards. The software was surprisingly easy to use, primarily because it is based on a knowledge and understanding of Excel. Also, if you ever have a chance to watch an Ignite presentation, you may want to do it. Ignite speakers have 5 minutes to talk and 20 presentation slides. The catch? The slides automatically forward every 15 seconds, whether or not the speaker is ready. The result is often a polished, focused summary of the presenter’s work or position. They were inspiring to watch!

Stella: I went to a presentation about college access programs that gave me lots to think about, like the different rubrics used to measure engineering programs. I also went to a session focused on environmental impact that expanded my horizons on what indicators to include for future opportunities. I was also impressed by the work of the University of Minnesota Extension’s cohort model for building evaluation capacity.

Joseph: One presentation I thought was particularly interesting was about evaluation in a for-profit, corporate environment. It ended up being more different [from nonprofit evaluation] than I could have previously imagined. The focus is more on process over outcome, with the understanding that process will lead to outcomes (new business, repeat business, etc.). I did learn some good takeaway strategies that furthered my process evaluation knowledge and skills that will be useful for any sector.

Dan: I found a couple of presentations particularly interesting. One was about using email and postal surveys (or survey invitations) together to maximize response rates, and it went through the different combinations [the presenter] used to deliver the survey (or invite people to complete the survey). Another session I attended was about the analysis of quantitative data with different software packages, and there were some that I knew of, but others I didn’t. I found the functionality of each different software package to be of interest.

ACET Staff

One of the things that I like best about evaluation is that there’s no specific formula to get the job done,  and although there are tried-and-true methods and best practices (like the National Institutes of Health’s Best Practices for Mixed Methods Research in the Health Sciences), evaluators have the ability to be flexible and work with stakeholders to design an efficacious evaluation that meets everyone’s needs.

Even as I enjoy the freedom to tailor evaluations to the needs of clients, I find it helpful to have a set of guidelines or a framework to fall back on. Because I’m trained in public health, my favorite framework – based on familiarity and ease of use – is the Centers for Disease Control and Prevention’s (CDC) Framework for Program Evaluation. The framework organizes the basic steps and standards of program evaluation into an easy to use form:

Image source: http://www.cdc.gov/eval/framework/index.htm

What I really like about this graphic is the illustration of the program evaluation process as a cyclical process, with the standards at the center of everything. The visual fits my personal beliefs about evaluation: that it should be ongoing and should always revolve around these set standards of practice.

As far as the listed standards, these groupings include 30 sub-standards set in place by the Joint Committee on Standards for Educational Evaluation.

I strive to keep them in mind throughout the evaluation process and while connecting with clients and program stakeholders:

1. Utility standards help to ensure that evaluation results will be useful to stakeholders, and that they fulfill the needs of users.
2. Feasibility standards are in place to help evaluators design an evaluation that is “realistic, prudent, diplomatic and frugal.”
3. Propriety standards concern legal and ethical aspects of evaluation and research, as well as ensure that the welfare of those involved in the evaluation (or who stand to be affected by the results) is protected.
4. Accuracy standards ensure that evaluation results will present an accurate picture of the program and its operations, especially any features that help to determine the “worth or merit” of the program.

While there’s no real benefit to using this framework over another, and many organizations develop their own framework or set of guidelines for evaluation, I find the CDC framework incredibly useful in guiding my evaluation work. Additionally, I believe that the focus on the standards of evaluation helps to promote transparency, which is a fundamental method of operation here at ACET.

What do you think of the CDC’s framework? Is it too simple? Too convoluted? Does it help you better visualize the evaluation process? We’d love to hear your reactions in the comments!

Elizabeth

In a previous blog post, I introduced the topic of Executive Summaries as well as a brief look at the keys to an effective summary. In this post, I want to identify the key components of the structure and flow of a summary.

Heading: The heading, or title, of your Executive Summary should be brief but still descriptive. It should communicate its function to its readers immediately.

First Paragraph: Introduce your client organization, its source of funding, the name of the program, its goals, and its target population.

Second Paragraph: Describe the program and evaluation specifics. This is a good place to include historical information about the project if relevant. Explain how the program was evaluated, the number of individuals who participated, to what extent, and how those contributions were measured.

Third Paragraph: Use your third paragraph to address impact statements. Describe the key findings for each evaluation goal previously identified.

Fourth Paragraph: Describe the strengths and barriers, or successes and challenges, encountered in the evaluation. This is also a good place to explore any other notable or unintended outcomes experienced.

Final Paragraph: This paragraph will serve as your conclusion. Use this paragraph to make recommendations or suggest next steps.

Beyond content, design is also a factor in delivering information in a clear and uncluttered way. If your Executive Summary consists of pages and pages of paragraphs, it is likely exceeding the target length of 1 to 2 pages and/or has been cut and pasted in large pieces from its original report. Keep in mind that the purpose of the Executive Summary is to succinctly report the most important aspects of an evaluation report to stakeholders, not simply to produce an abridged version of the original report. Try using visually distinct headers to make the layout of the information clear (similar to the above bold words). Also, to only report the most important information, leave out charts, graphs, and tables in the construction of your Executive Summary; readers can find these components in the full report if they want further information.

Good luck.

Stella

The Kresge and Lumina Foundations have announced “Credit When It’s Due: Recognizing the Value of Quality Associate Degrees,” a program intended to address the issue of students who transfer from community colleges to 4-year institutions prior to receiving associate degrees, but with the goal of earning 4-year baccalaureate degrees. Citing a lack of systems for community colleges to log credits earned after transfer, the program is meant to encourage the implementation of methods that will award associate degrees to transfer students who have earned the credit, which are commonly known as “reverse transfer” or “reverse back” degrees.

Per the announcement, which can be read here, the grant is available to state higher-education executive offices, state systems of higher education representing 2 and 4-year institutions, 4-year institutions, or community colleges, and partnerships of 2 and/or 4-year institutions with a designated lead public postsecondary institution or nonprofit organization as the grant manager. It is expected that 8 to 10 grants will be awarded in amounts between $200,000 and $600,000 with 2-year durations, based on the number of community college/university partnerships needed to scale the project. Only one award will be made per state, and each project will be required to contribute to cost-sharing at the state level. For the full RFP, visit the Lumina Foundation’s website.

The Early Childhood Personnel Preparation (ECPP) project is a statewide study assessing the capacity and effectiveness of Minnesota’s 2- and 4-year institutes of higher education to prepare early childhood educators and care providers. Minnesota is one of the first states to seek to define effectiveness of early childhood higher education programs and to identify gaps in data that is currently collected. In addition, data collected throughout the ECPP project will inform further investigation of the capacity and effectiveness of Minnesota’s institutes of higher education and will likely be a model for other states. ACET is managing the ECPP project in collaboration with multiple state agencies (Minnesota’s Departments of Education, Human Services, and Health) and ten advisory group members.

Further information on the ECPP project can be found on ACET’s website: (http://www.acetinc.com/ECPP/AG.htm), and includes:

• ECPP advisory group and state agency members;
• Advisory group meeting agendas and notes;
• Evaluation framework;
• Online resources (databases and directories); and
• Proposed uses for the study findings.

Coming in June 2012 you will find the technical reports outlining our findings!

If you have any questions about the project, feel free to contact Dan or Joseph at 952.922.1811.

Dan

Not too long ago my husband Zack and I celebrated our 5^th wedding anniversary and we decided to splurge on a shared gift – a new iPad 2. It’s been a great tool for both of us – the iPad is incredibly portable, which comes in handy when I have a meeting with a client, and Zack has been using the iPad nearly non-stop during NaNoWriMo (National Novel Writing Month).

Initially, we had no efficient way to keep everything on the iPad synched with our desktop computer (at home) and my desktop computer at ACET. But I was recently introduced to two great – and free – tools that can help anyone with multiple computers keep their files synched and available anywhere. They are Dropbox and Evernote.

Dropbox (http://www.dropbox.com/) provides an online storage point that can be accessed anywhere. And you can link that storage point to multiple computers for easy access to files. Once you install the program on your computer, any file you store in Dropbox becomes available on other computers linked to your account. So I can put a file in to Dropbox at work and see the file later at home, without using a flash drive. I can also go to a client’s office, log in to my Dropbox account, and show my client any relevant files. Dropbox also has an easy-to-manage file sharing system. I can set up a folder in my Dropbox and share the contents of that folder with colleagues in another part of the country or the world. File sharing through Dropbox is quite secure – when I invite someone to share a folder in Dropbox my colleague can only see the shared folder and does not see the other files I have stored in Dropbox. Dropbox is also very secure and uses HIPAA-level encryption to ensure your files are safe. And Dropbox is available in multiple platforms: Windows, Macs (including iPhones and iPads), Linux, Android, and Blackberry. So I can use my Windows computer at ACET and share files with my Mac at home! Although Dropbox does have a limit on the amount of information you can store on their servers (2GB for free), you can upgrade for a fee.

While Dropbox is a great resource for storing and sharing files, you don’t actually create any file in Dropbox. That’s where Evernote (http://www.evernote.com/) comes in. Like Dropbox, Evernote works across multiple platforms including Windows, Mac, several smart phones, and several mobile tablets. And, like Dropbox, the Evernote software provides online storage for a wide range of file types – like documents, PDFs, images, MP3s/MP4s, videos – that can be accessed from computers or mobile devices when you log in. But unlike Dropbox, Evernote’s forte is that you can create documents in Evernote, especially notes. Evernote users can construct to-do lists, typed and handwritten (using a mouse, stylus, or fingers) notes, and audio or webcam recordings. You can even grab part or all of a web page and store in on Evernote. Evernote also has an incredibly powerful search feature – it will search titles and contents of documents, search through file tags, and will even search handwritten notes to find the word or phrase you are looking for. You can share the contents of Evernote with others, but that function is less transparent and requires more time to manage than does Dropbox. Last, Evernote has basic services for free (60MB of storage space), but you can upgrade for a fee.

Do you use any software that helps your productivity? If so, please share!

Kirsten

Here is a link that highlights a variety of webinar hosts for small businesses.  This is a great place to start if you’re looking to add webinars as an option in your program.

Hello and happy Friday!

Just wanted to pass on a wonderful resource to those of you involved in statistical analysis. Click here for a fantastic compilation of links to FREE software!

This list was put together by a staff member of Harvard University.

Enjoy!

Last week, we posted about document sharing resources that organizations may find useful. This week, we would like to follow up with a post sharing online resources that you may appreciate.

Tagxedo is a free program that creates tag clouds, also known as word clouds, in visually-appealing images and styles. This resource can be particularly useful in helping to analyze and display qualitative data. Users simply copy and paste all their prose into the text box. From there, the program automatically removes common words such as “and” or “so,” and allows people to select words from a list that they want to include. Users can choose an image that best depicts the text subject(s) or upload an image of their own. They can also create tag clouds from a blog, a Twitter ID, a news story, or a variety of other online sources by uploading a link rather than pasting in text. The program will require you to install Microsoft Silverlight in order to run, but the program is free, simple, and entirely safe. See below for the poem “Peanut-Butter Sandwich” by Shel Silverstein.

Prezi is a free web-based presentation application that uses a single page instead of multiple slides. All information (i.e., text, images) is put on one canvas and then grouped into frames. This allows the user to create a nonlinear “story” of sorts. The presenter can create a path for the frames or choose to select a section at will. Although the application is web-based, the user can download the presentation at any time if they don’t have an internet connection. The basic version of Prezi is free to use, but must be published on the Prezi site; however, private licenses can be purchased.

Another presentation tool, MindMeister, allows users to create a mind map to show information. Users can change font size and color, insert images or icons, add links, and rearrange spaces as needed. The map fits as many paths as you need for any given area, automatically readjusting to make space. Each caption and path is easy to create – click “add” to create a new path, “delete” to remove an unwanted path, or move a path by clicking on the caption and dragging it to a different path. MindMeister does only allow three free maps before you have to buy a membership to use the program, but the membership is low-cost at $4.92 per month. See below for my example that, again, highlights peanut butter sandwiches.

What tools or programs do you find most useful in your work? Leave a comment below and let us know!

Cassie

When you want to share information, either hosting a presentation or simply wanting to share work on a document, what resources do you use? We’ve found some particularly useful tools for sharing and presenting information (click on the link attached to the application to view more about that resource). This week we’re posting on tools for sharing documents.

A simple way to share documents that are not confidential is by uploading them to an online space. By using an online space, users are able to access the document anywhere they have internet access.

Google offers a free online sharing application called Google Documents that allows users to not only share documents, but also work on a document at the same time as other users. Once uploaded, users can edit, add, and change information in a document much like they can in Microsoft Word or a similar text editing software. When one user uploads a document, they can share with other users who can view or edit those documents. The original user must have a Google account to upload the document, but any other users do not need a Google account.

Dropbox is also a free project collaboration tool that allows users to upload and store files on multiple spaces of their choice. Just as with Google Documents, users can upload documents, photos, and more. An additional feature that Dropbox offers is the ability to link the application to multiple computers and phones. The program creates folders on each computer or phone that a user wants, allowing a safe space to upload documents without having to go onto a website.

Stay tuned for next week’s post on online resources!

Cassie