SYLLABUS
UST BC 3200 PROGRAM EVALUATION METHODS
Fall 1998
William McAllister
People in society organize efforts to try to affect theirs or others’ circumstances. We initiate educational efforts, health efforts, housing, psychological, economic and other kinds of efforts. These efforts we can call “programs”, such as anti-smoking, alcohol abuse and illegal drug treatment programs; programs to give money to poor people or to house homeless people; programs to combat air and water pollution, and so forth. An important premise of this course is that such programs ought to be studied to try to answer questions such as: Should a program be set up in the first place? What kind of program should that be? Whom or what should it target? Was the program implemented as designed? How was implementation accomplished? Did the program achieve what it was supposed to achieve? How did it do that, and what was the cost? Or why didn’t it accomplish its objectives? And what unexpected benefits, harms or other consequences occurred?
These are the kinds of questions that the methods of program evaluation try to answer, and this course will introduce you to some of those methods. This course focuses on methodology, on how you go about learning something; specifically, how you go about answering these kinds of questions about social welfare, environmental, health and other programs. We are less interested in “answers” to questions than in how you arrive at an answer.
To do this, we will organize ourselves into two-person research groups that will each develop an evaluation design for an actual program. Class time will be spent figuring out what is needed to construct evaluation designs and understanding why these things are necessary. We will do this by solving problems researchers confront in creating designs through readings from evaluation textbooks, actual designs, published reports of actual evaluations and articles from the mass media, and by learning statistical analysis techniques such as univariate statistics, difference of means tests and ordinary least squares regression analysis and a computer program for carrying out these analyses. We will also have guest speakers—a program person and a researcher—from a program that has used evaluation techniques.
Readings. The principal readings are from two books: Evaluation: A Systematic Approach by Peter H. Rossi and Howard E. Freeman, 5th edition, and By Design: Planning Research on Higher Education, by Richard J. Light, Judith D. Singer and John B. Willett. Both can be purchased at Labyrinth Bookstore. Supplementary readings will be available at the Lehman Library reserve desk, at the Lehman Computer Lab (computer manuals), or will be handed out in class.
Structure. The course is for three credits and meets from 4:10 to 5:25 Mondays and Wednesdays. Class time will be spent as a group figuring out what’s involved in carrying out different kinds of evaluations and will rely on student’s leading the conversation. Time will also be spent in the computer lab, both during the class and outside, and in working with actual programs for which a design is being developed. As part of the exchange for providing resources to us, you may volunteer 10 hours of work at these programs over the course of the semester.
Grades. Grades will be based on the following:
- presentation of articles from mass media sources,
- leading conversation in the classroom,
- general participation in class, e.g., asking questions,
- three homework problem sets,
- in-class mid-term and final exams, and
- an evaluation design.
Course Outline
Except for the Introduction, the outline does not necessarily reflect the order in which the course will proceed. We may change the order of topics—and so do the appropriate reading—as different methodological issues arise in our class conversations about what you next need to know to create an evaluation design. The attached sheet gives likely dates for each topic, if we were to keep to the ordering of topics given here.
Topics are in bold
R & F refers to Rossi and Freeman
L, S & W refers to Light, Singer and Willett
Opening day
Handout: Example of program evaluation study from mass media article
Introduction
R & F: Chapter 1. Programs, Policies and Evaluations
R & F: Chapter 10. The Social Context of Evaluation
Ethics in program evaluation
Handouts: “Ethics in Program Evaluation” by Emil J. Posavac and Raymond G. Carey
Examples of Participant Consent forms;
Mass media articles on actual ethical dilemmas in current research
Needs assessment
R & F: Chapter 2. Diagnostic Procedures
L, S & W: Chapter 2. What Are Your Questions?, pp. 12-23
Handouts: “Mapping Community Capacity” by John L. McNight and John P. Kretzman;
Example of a request for needs assessment and response to request;
Example of a needs assessment report;
Mass media article on utility of needs assessment
Logic of impact / outcome evaluation
R & F: Chapter 5. Strategies for Impact Assessment
L, S & W: Chapter 5. Compared to What?
Handouts: Rossi & Freeman design logic formulations;
Report on study of job training program
Measurement theory
L, S & W: Chapter 6. What Are Your Outcomes?, pp. 150-160, and
Chapter 7. How Can You Improve Your Measures?
Handouts: Mass media articles on measurement issues;
Statistical effects of unreliability
Guest speakers: How program directors and evaluators see and work with each other
Mid-term exam: Wednesday, 21 October 98
Quantitative analysis: Measurement in practice
Handouts: SPSS for Windows Introductory Guide, pp. 1-8, 11-12;
Descriptions of Memphis New Mothers study and data
NOTE: Class meets in Altschul computer lab, 4th floor
HOMEWORK 1: Basic data file manipulations in SPSS
Quantitative analysis: Univariate descriptive statistics
Handouts: Calculating univariate statistics;
SPSS for Windows Introductory Guide, p. 12
NOTE: Class meets in Altschul computer lab, 4th floor
HOMEWORK 2: Descriptive statistics using SPSS
Quantitative analysis: Differences of means
Handouts: Logic of hypothesis testing;
Calculating difference of means;
SPSS for Windows Introductory Guide, p. 14
NOTE: Class meets in Altschul computer lab, 4th floor
Initial draft of evaluation design due: Wednesday, 4 November 98
Quantitative analysis: OLS regression
Handouts: Calculating OLS regression;
SPSS for Windows Introductory Guide, p. 18
NOTE: Class meets in Altschul computer lab, 4th floor
HOMEWORK 3: Difference of means and ols regression using SPSS
Designs for impact / outcome evaluation
R & F: Chapter 6. Randomized Designs for Impact Assessment
R & F: Chapter 7. Quasi-experimental Impact Assessments
R & F: Chapter 8. Assessment of Full-Coverage Programs
Handouts: Overview of outcome evaluation methodology;
Experimental designs: rationale and examples;
Non-experimental, comparative observational designs: rationale and examples
Fiscal evaluation
R & F: Chapter 9. Measuring Efficiency
Handouts: Excerpts from “A Benefit-Cost Analysis of the Supported Work Experiment” by Peter Kemper, David A. Long and Craig Thornton;
“Research Note: Using Cost-Benefit Analysis to Measure Rehabilitation and Special Deterrence” by Tara Gray;
Mass media / excerpts from professional articles on valuation and ethics
Qualitative analytic techniques
Handout: “Qualitative Evaluation Methods” by Emil J. Posavac and Raymond G. Carey
Final evaluation design due: Friday, 11 December 98
Final exam: Monday, 14 December 98
|