Lake Tahoe Collaborative Annual June Forum & Retreat

June 10, 2013


1. 12:00pm Lunch        

2. 12:15pm Welcome & Introductions, Wendy David, Facilitator of LTC

3. Letter of Support discussion between LTC and Barton Community Clinic, Caryn Mahoney

Barton Community Clinic is reapplying for the Title X grant. Barton Community Clinic would like to have the LTC sign a Letter of Support. Title X will provide Pap Smears, Breast Exams, reproductive care for the community. The Clinic will also partner with LTC members on client referrals for programs and services. They offer a sliding scale for service for uninsured and underinsured. An outreach element, specifically for adolescents, is part of this grant.  The LTC agreed to sign the Letter of Support.

4.   12:30pm Elizabeth Christiansen, Ph.D.

Director, Center for Program Evaluation 

University of Nevada, Reno

(775) 682-6853

“Planning an Effective Program Evaluation” 

Elizabeth’s background is in social psychology. She obtained her PhD from the University of Nevada at Reno. The interdisciplinary program appealed to her because of the overlapping and integrated services. She likes applied research, got a graduate assistantship, then a faculty position. She has studied social psychology, youth victimization, equality and gender issues.

The Center for Program Evaluation has been around since before 1999. They take a partnership approach to evaluation. They recognize the expertise that people who run a program know about it, and then they work with an evaluation based on the funder needs and the program needs.

What is Evaluation?

“Program Evaluation is the systemic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve the program effectiveness, and or inform decisions about future programming.”  -Michael Quinn Patton.

What is an Evaluation Plan?

An Evaluation Plan is a document that describes how you will monitor and evaluate your program and how you intend to use evaluation results from program improvement and decision making.

It’s a good idea to put information on paper about where you are going and what you are doing.

Writing out a plan is great at making sure all people are on the same page, they all know the reasons for doing things, and that everyone can look back and track and make sure they are doing what the program intended to do.

Evaluation Planning Steps:

1. Identify and engage stakeholders

Who is involved in your program? Who is affected by your program? You want to engage stakeholders to get their buy-in. This increases the buy-in and support. They can help you collect data. You may develop an evaluation team of staff or stakeholders.

An evaluation team makes sure the evaluation meets the needs of the stakeholders,

increases the credibility of analysis and interpretation, and helps ensure the evaluation results are used.

2. Identify the purpose of the program.

Why are you doing the evaluation?

Identify the purpose of your evaluation:

For accountability?

To make improvements?

To generate knowledge? Are you in a demonstration program and how do you describe this knowledge to your field and how is it applicable to other programs?

Types of evaluation:

Formative (process) occurs during program planning and implementation and is conducted for the primary purpose of program improvement. How are our systems running? What can we do to make things more efficient? Good to have a third outside party look at this. Journaling and reflective processes with staff work well.

Summative (Outcome) occurs towards the end of the implementation of a program and is conducted for the purpose of accountability, increasing support and funding, and sustainability. Is the program effective in doing what we wanted it to do?

3. Describe the program and desired outcomes, The Logic Model

What is happening in the program?

Write a statement of need: Why does this program need to exist? What is the situation here in the community and why do we need this program here?

Develop a logic model: Inputs: available resources; outputs: program activities; outcomes: desired results of the program (short term, intermediate, long term)

This can be revisited to show you where to go next, shows all resources available (time, staff, money). You can show this to a funder/staff.

An example of Logic Model could include a flow chart with a description of the: Situation, Priorities, Inputs, Outputs, Outcomes, Assumptions, External Factors, and Evaluation

Assumptions and External Factors

Evaluation: You should be able to measure your outcomes

The University of Wisconsin Extension has a great Logic Model outline online, or look online for others.

4. Develop evaluation questions.

What do we want to find out about the program?

It’s useful to have stakeholders involved in this process.

What do the various stakeholders want to know about the program?

You need to prioritize the questions to get the key questions answered as there is usually not enough money to answer all the questions.

Relate the questions generated to the main evaluation purpose(s)

Prioritize the questions.

5. Determine methods

How are you going to measure the results?

Select appropriate methods for the questions and the program.

Consider the types of questions you are asking, what data sources are available, and who will collect the data.

Surveys are highly used, but not always effective.

Data Sources:

Existing Data (Secondary):

Anytime you can avoid asking another question, this is desired.

Attendance records

Intake forms: demographics, etc.

Staff or participant journals

Primary Data Collection:

Program participants

Program administrators

Program staff

Program collaborators/partners

Community members


Surveys: satisfaction with program and pre-post tests

Individual interviews

Focus groups




6. Plan how the results will be analyzed, interpreted, reported, and used.

How will you understand and use the results?

Interpret the results.

What conclusions can we draw from these data? What claims can you make about your program based on these results?

To what extent do the findings support your expectations?

What are the possible explanations for the results?

This is another great time to have the stakeholders involved and to have a facilitator involved.

Sharing the results:

Tailor the reporting to the various audiences. People like to know what you are doing (community, staff, funders).

Consider different forms of reporting

Use evaluation results in staff and stakeholder meetings


Audience Questions:

Q: If someone is doing evaluations, what services does your Center provide?

A: If someone knows they are writing a grant, they can write us in the grant and we can help with the evaluation. We can assist an agency with program evaluation. We work in Nevada and California. There is a fee for the services.


Q: Are you working with students? Internships?

A: We work with Masters of Public Health students. Student internships are available each year. There are field studies classes that assist in agency program evaluation. Elizabeth advises students but does not teach classes.


Q: A reference to Physiologic Responses was in a past slide, please explain this.

A: These are mostly health related, example: height or weight.


Q: Are you also noticing more of a demand from funders that evaluation has to be a key part of funding? American Evaluation Association has great resources and some great books recently written about evaluation.

A: Yes, accountability is getting steeper. Some grants require a certain amount of money to be put toward evaluation. There are some experimental and quasi-experimental programs, so some small communities require more assistance with this.



5. Looking Forward to 2013-14, Wendy David


Issues/Topic to explore in 2013/2014:


a. “Safe Place”—what are ways we can communicate this on the busses? Papers?

-Identify community resources: NAMI, Mental Health, Live Violence Free, and promote crisis community resources. TYFS can do the design, what numbers do we include and what information do we include?

-Develop collaborative promotion (busses, maintain consistency)


b. Shelters/Warm Room

-Place for homeless to gather to escape weather

-Target population: elderly, income, mental health (bring community together)


c. LTC agencies to sign an MOU. Then all member agencies will have this document to submit for grants.

-Invite decision makers to a meeting and what does it take to get this signed? Edited then signed? Need an MOU for a program and that is effective.

-Nonprofit get a standing agenda item once/year to talk about issues and updates.

-Formal demonstration of support through MOUs and regular renewal

-What information is needed to support an MOU/identify areas of need?

-MOU needs to be sustainable and program directed

-Part of being a LTC member is to sign an MOU that lists each agency and their responsibilities, financial responsibility.

-Two levels of MOUs: one is partnership MOU; find a template for a formative agreement on certain topics. Other is financial



Meeting Educational/Presentation Topics:

Children Protection Unit-El Dorado County

Father Engagement (CapC)

Montessori Principles in Eldercare—Pat Okacza

Frank Bigelow and Ted Gaines—Angela will facilitate

Fall update from Foster Family and Adoptive Services

Dental Update

Parent Leadership


6. 1:45pm Adjourn


7. No meeting in July. Our next meeting is August 12, 2013.

     Boys & Girls Club Meeting Room, 1100 Lyons Avenue


8.  No June Drug-Free Community Coalition Meeting.