Program Evaluation-Part 1 CITAR Workshop July 28-Aug 1, 2008 Hue City

Definition of program evaluation

• “…the systematic collection of information about the activities … and outcomes of programs
• tomakejudgments about the program,
• [in order to]improve program effectiveness, and/or informdecisions about future program development” (Patton)

Part 1: Objectives Participants will be able to

• Define program evaluation
• Describe 2 sources of failure of a program to achieve a population impact
• Distinguish the questions, methods, and standards of comparison of the 1^st level of evaluation from those above it in the hierarchy:

1) concept and design evaluation

2) process evaluation

3) outcome evaluation

4) economic evaluation

Potential Sources of Failure to Achieve a Population Impact

Concept & Design Evaluation

• Step 1, Describe the program or policy
  • Problem it is supposed to solve
  • “Target” population
  • Objectives
  • Program theory – Methods
  • Delivery strategies
  • Resources required
  • Stage of development
  • Organizational and political context
• Step 2 – Assess the need for the program
  • What is the evidence that the problem is “real” and worthwhile trying to solve or reduce?
  • Are the appropriate population(s) being targeted?
  • What standards of comparison might be used here?
    • Expectations of funders
    • Expert opinion
    • Social justice
• Step 3 - Assess the program theory
  • Do the change objectives reflect validassump-tionsaboutthecauses of the problem?
  • Are the interventionmethodsappropriate?
  • Will the planned deliverystrategies reach the target population?
  • Are the planned resourcesadequate to support the program and its delivery?

What standards of comparison might we use?

• Expert opinion Past achievements Theoretical standards
• Norms Professional standards

Use a logic model

• A systematic, visual way to present and communicate an understanding of the relations among the resources for operating the program, the activities that comprise the program, and the intended outcomes
• Shows how the program is supposed to reach the outcomes
• Provides the basis (logic, theory, evidence) for expecting these outcomes

Basic Logic Model Template

Inputs

What resources are needed for this program to operate?

• Funding
• Personnel
• Materials
• Equipment
• Relationships
• Others?

Activities

• What events or actions are planned to take place to create the desired results?

Outputs

Impact of the program

• Short-term
• Intermediate
• Long-term

Tips for Writing Logic Models

• Be specific

Knowledge of what? Which beliefs? skills?

• Work both left and right

Even if you start with a program, specify the problem at the far left before you start to work back and forth between them.

• Modify the form – columns

You might need more categories to convey the causal chain of hypotheses.

Approaches to developing a logic model

• No right way
• Clear definition of problem should be done before starting
• Forward (left to right)
  • Ask the question “But why?”; or
  • If…. Then….
• Reverse-
  • Starts with desired outcomes
  • Asks the question, “But How?”
• Modify the form - rows Use rows & partial rows to separate causal chains; e.g., unintended pregnancy & STDs don’t belong on the same row if main intervention is oral contraceptives
• Recognize differing perspectives

Usually there are >1 possible logic models for the same problem

Logic Model: Source of Causal and Intervention Hypotheses

• Causal Hypothesis: What is the program’s underlying hypothesis about the “cause(s)” (or determinants, risk factors) of the problem?
• Intervention Hypothesis: What are the program’s underlying hypotheses about the “cause(s)” of the behavior and what will change those “causes”?
  • The “Logic” of the Causal Hypothesis

The logic or rationale for a program’s causal hypothesis should be developed from:

• Epidemiological evidence
• Theoretical evidence
• Evidence from other programs
• Qualitative data

Causal Hypothesis: Examples

Unprotected sex among adolescent girls

What are some of the “causes” (i.e., risk factors or determinants)?

• Too much sex in the media
• Girls don’t know the risks
• Condoms are not available
• Girls want to get pregnant
• Welfare supports pregnancy as a career
• Poverty supports early pregnancy as normative

The “Logic” of the Intervention Hypothesis

What is the evidence for this intervention hypothesis?

• Focus groups
• Key informant assessment
• Formal behavioral theory
• Consensus-based theory
• Previous evaluation results
• Evidenced-based guidelines for practice
• “Intervention Mapping” is one approach which uses theory and evidence to develop an intervention program

Intervention Hypothesis: Examples

• Too much sex in the media
  • Movie ratings, other controls reduce media exposure
  • Promotion of role models who don’t have sex
• Girls don’t know the risks
  • Provide information about the risks by peers
• Condoms are not available
  • Make condoms more accessible to girls
  • Boys?
• Girls want to get pregnant
  • Promote alternative norms for relationships
  • Provide experience with the responsibilities of parenting
• Welfare policy supports pregnancy as a career
  • Reform welfare policy
  • Promote plausible “futures” for girls
  • Improve schools
• Poverty causes girls to see early pregnancy as normative
  • Mixed income housing

This afternoon’s exercise

1. What problem is addressed?
   • What evidence is presented that this problem affects the groups “targeted” in this study?
2. What are stated or inferred causal Ho?
   • What evidence is presented?
3. What factors/”determinants” are addressed?
   • What evidence is presented?
4. What are stated or inferred intervention Ho?
   • What evidence is presented?

1. How well would these causal hypotheses fit Vietnam?
   • What evidence would support your opinion?
2. How well does the profile of factors or ‘determinents fit Vietnam?
   • What evidence would support your opinion?
3. How well would these intervention hypotheses fit Vietnam?
   • What evidence would support your opinion?

Click here to download slide powerpoint