Program Evaluation-Part 2 CITAR Workshop July 28-Aug 1, 2008 Hue City

Objectives

• Distinguish 4 levels of evaluation:

1. concept and design evaluation ☑
2. process
3. outcome
4. economic

• Show understanding of the process evaluation level by matching evaluation questions and standards of comparison

Methods

• Compare the plan for who is supposed to do what to whom, when, how, and where and with what resources to
• With what actually happens
• Can also consider other standards of comparison
• Think regular audits, management information systems with short feedback loops oriented toward improving the quality of program processes

Process Evaluation Domains

• Reach
• Fidelity, operationalization of key intervention elements
• Resources, cost assessment
• Process Evaluation Reach

Impact as a function of reach + efficacy

Efficacy x Participation = effectiveness

* Individual/face-to-face:

30% x 5% = 1.5%

* Local media in the community:

6% x 75% = 4.5%

Why do you think this program did not show a treatment effect?

Sources of coverage bias

• Self-selection, appeal of the program
• Selection by program staff, referral sources, others
• Methods of recruitment and retention
• Other influences, e.g., unexpected barriers and facilitators (e.g., hours of program services, types of providers)
• Biased or perceived-to-be-biased channel or delivery system

Process Evaluation Fidelity of Delivery

• How faithful is the program that is [actually] delivered to the program that was planned?
• What elements are missing? How important are they? Why are they missing?
• If this is a multi-layer intervention, are preliminary steps in the causal chain at the lowest level enacted?

Fidelity Assessment - examples

• Quality of services delivered
  • Content, implementation of intended methods?
  • Facilitator characteristics, type of interaction, style
• Time
  • Total, distribution across content or tasks and contact
• Staffing
  • Qualifications, characteristics, skills, experience, performance of intended role

Process EvaluationFidelity of Reception

• What elements of the program are actually received by the target group compared to what was planned?
• What elements are not received? How important are they? Why are they not received? Where did they come from?
• Are critical features of the intervention perceived by the target group(s) as they were intended? Why not?

Process Evaluation Resources

• What resources and other organizational functions are in place to support the program compared to what was planned?
• What resources are missing? How important are they? Why are they not available?
• Are additional resources available to support the program? What is/are their source(s)?
• What is the cost of this program? (to be covered under economic evaluation)

Click here and here to download slide powerpoint