| Preface |
|
xiii | |
| PART I: BUILDING BLOCKS OF EVALUATION |
|
|
|
|
3 | (30) |
|
The Quality Improvement Process |
|
|
7 | (1) |
|
Why Should We Do Evaluations? |
|
|
8 | (6) |
|
Increase our Knowledge Base |
|
|
8 | (2) |
|
|
|
10 | (3) |
|
Demonstrate Accountability |
|
|
13 | (1) |
|
Assure That Client Objectives Are Met |
|
|
14 | (1) |
|
|
|
14 | (2) |
|
|
|
16 | (1) |
|
Quality Improvement Approaches |
|
|
17 | (8) |
|
|
|
17 | (3) |
|
|
|
20 | (5) |
|
|
|
25 | (2) |
|
Summing Up and Looking Ahead |
|
|
27 | (1) |
|
|
|
28 | (1) |
|
|
|
29 | (1) |
|
References and Further Readings |
|
|
30 | (3) |
|
|
|
33 | (26) |
|
|
|
33 | (4) |
|
Agency Mission Statements |
|
|
34 | (1) |
|
|
|
35 | (1) |
|
|
|
36 | (1) |
|
|
|
37 | (5) |
|
An Agency Versus a Program |
|
|
42 | (1) |
|
|
|
42 | (1) |
|
|
|
43 | (4) |
|
Unintended Program Results |
|
|
45 | (1) |
|
Program Goals Versus Agency Goals |
|
|
45 | (2) |
|
Types of Program Objectives |
|
|
47 | (1) |
|
|
|
47 | (1) |
|
|
|
48 | (1) |
|
|
|
48 | (1) |
|
Qualities of Program Objectives |
|
|
48 | (3) |
|
|
|
49 | (1) |
|
|
|
50 | (1) |
|
|
|
50 | (1) |
|
|
|
51 | (1) |
|
Program and Practice Objectives |
|
|
51 | (2) |
|
|
|
53 | (2) |
|
Program Logic Model: An Example |
|
|
55 | (1) |
|
Summing Up and Looking Ahead |
|
|
55 | (1) |
|
|
|
56 | (1) |
|
|
|
56 | (1) |
|
References and Further Readings |
|
|
57 | (2) |
|
|
|
59 | (30) |
|
|
|
60 | (7) |
|
Visibility of Social Problems |
|
|
61 | (1) |
|
Wants, Demands, and Needs |
|
|
61 | (1) |
|
Perceived and Expressed Needs |
|
|
62 | (1) |
|
Need Assessment Questions |
|
|
63 | (4) |
|
|
|
67 | (2) |
|
|
|
67 | (1) |
|
|
|
68 | (1) |
|
Evaluability Assessment Questions |
|
|
69 | (1) |
|
|
|
69 | (4) |
|
Formative and Summative Evaluations |
|
|
70 | (1) |
|
|
|
71 | (2) |
|
Process Evaluation Questions |
|
|
73 | (1) |
|
|
|
73 | (6) |
|
|
|
75 | (1) |
|
|
|
76 | (1) |
|
|
|
77 | (1) |
|
|
|
78 | (1) |
|
|
|
79 | (5) |
|
|
|
81 | (1) |
|
Valuing Human Experiences |
|
|
82 | (2) |
|
|
|
84 | (1) |
|
Summing Up and Looking Ahead |
|
|
84 | (1) |
|
|
|
85 | (1) |
|
|
|
86 | (1) |
|
References and Further Readings |
|
|
86 | (3) |
|
Planning and Focusing an Evaluation |
|
|
89 | (24) |
|
Evaluation as Representation |
|
|
90 | (2) |
|
Common Characteristics of All Evaluations |
|
|
92 | (3) |
|
|
|
92 | (1) |
|
|
|
93 | (1) |
|
|
|
94 | (1) |
|
|
|
94 | (1) |
|
|
|
95 | (1) |
|
Presentation of Evaluation Findings |
|
|
95 | (1) |
|
|
|
95 | (2) |
|
Planning with Stakeholders |
|
|
97 | (1) |
|
|
|
98 | (2) |
|
Selecting What to Monitor |
|
|
100 | (8) |
|
|
|
100 | (2) |
|
|
|
102 | (1) |
|
|
|
103 | (2) |
|
|
|
105 | (1) |
|
|
|
106 | (2) |
|
Summing Up and Looking Ahead |
|
|
108 | (1) |
|
|
|
108 | (1) |
|
|
|
109 | (1) |
|
References and Further Readings |
|
|
110 | (3) |
| PART II: TOOLS OF EVALUATION |
|
|
Data Sources, Sampling, and Data Collection |
|
|
113 | (30) |
|
|
|
114 | (2) |
|
|
|
116 | (4) |
|
|
|
120 | (12) |
|
|
|
120 | (5) |
|
|
|
125 | (7) |
|
Fitting Data Collection to the Program |
|
|
132 | (3) |
|
|
|
132 | (2) |
|
Appropriateness to the Flow of Program Operations |
|
|
134 | (1) |
|
|
|
135 | (1) |
|
Developing a Data Collection Monitoring System |
|
|
135 | (1) |
|
Summing Up and Looking Ahead |
|
|
136 | (1) |
|
|
|
136 | (3) |
|
|
|
139 | (1) |
|
References and Further Readings |
|
|
140 | (3) |
|
|
|
143 | (36) |
|
Why Measurement is Necessary |
|
|
143 | (3) |
|
|
|
145 | (1) |
|
|
|
145 | (1) |
|
Types of Measuring Instruments |
|
|
146 | (12) |
|
|
|
147 | (3) |
|
|
|
150 | (4) |
|
Goal Attainment Scaling (GAS) |
|
|
154 | (4) |
|
Standardized Measuring Instruments |
|
|
158 | (2) |
|
Locating Standardized Measuring Instruments |
|
|
160 | (4) |
|
|
|
160 | (2) |
|
Professional Journals and Books |
|
|
162 | (2) |
|
Evaluating Measuring Instruments |
|
|
164 | (9) |
|
|
|
165 | (1) |
|
|
|
166 | (1) |
|
|
|
167 | (1) |
|
|
|
167 | (1) |
|
Representativeness of the Sample |
|
|
168 | (1) |
|
|
|
168 | (5) |
|
Summing Up and Looking Ahead |
|
|
173 | (1) |
|
|
|
173 | (1) |
|
|
|
174 | (1) |
|
References and Further Readings |
|
|
175 | (4) |
|
|
|
179 | (30) |
|
Informal Case-Level Evaluations |
|
|
179 | (2) |
|
|
|
180 | (1) |
|
|
|
180 | (1) |
|
Formal Case-Level Evaluations |
|
|
181 | (5) |
|
|
|
182 | (1) |
|
Measurable Practice Objectives |
|
|
182 | (1) |
|
|
|
183 | (1) |
|
|
|
183 | (3) |
|
|
|
186 | (1) |
|
Evaluation Design Continuum |
|
|
186 | (12) |
|
|
|
187 | (3) |
|
|
|
190 | (4) |
|
|
|
194 | (4) |
|
Advantages of Case-Level Designs |
|
|
198 | (3) |
|
The Practitioner Is Responsible for Data Collection |
|
|
199 | (1) |
|
The Focus Is on the Client |
|
|
199 | (1) |
|
Clinical Decisions Are Based on Data Collected |
|
|
200 | (1) |
|
Problem Analysis Is Facilitated |
|
|
200 | (1) |
|
The Cost Is Small In Time and Disruptiveness |
|
|
200 | (1) |
|
Client's Situation Is Taken into Account |
|
|
201 | (1) |
|
Data Can Be Used in Program-Level Evaluations |
|
|
201 | (1) |
|
Limitations of Case-Level Designs |
|
|
201 | (2) |
|
|
|
202 | (1) |
|
Explanatory Designs Are Difficult to Implement |
|
|
202 | (1) |
|
Summing Up and Looking Ahead |
|
|
203 | (1) |
|
|
|
203 | (1) |
|
|
|
204 | (2) |
|
References and Further Readings |
|
|
206 | (3) |
|
Program-Level Evaluations |
|
|
209 | (48) |
|
|
|
209 | (2) |
|
|
|
210 | (1) |
|
|
|
210 | (1) |
|
|
|
210 | (1) |
|
|
|
211 | (1) |
|
Characteristics of ``Ideal'' Evaluations |
|
|
211 | (9) |
|
Time Order of the Intervention |
|
|
212 | (1) |
|
Manipulation of the Intervention |
|
|
213 | (1) |
|
Relationship Between Interventions and Objectives |
|
|
214 | (1) |
|
Control of Rival Hypotheses |
|
|
214 | (6) |
|
Internal and External Validity |
|
|
220 | (7) |
|
Threats to Internal Validity |
|
|
220 | (6) |
|
Threats to External Validity |
|
|
226 | (1) |
|
Program-Level Evaluation Designs |
|
|
227 | (2) |
|
|
|
229 | (8) |
|
One-Group Posttest-Only Design |
|
|
230 | (2) |
|
Multigroup Posttest-Only Design |
|
|
232 | (1) |
|
Longitudinal Case Study Design |
|
|
233 | (1) |
|
Longitudinal Survey Design |
|
|
234 | (3) |
|
|
|
237 | (7) |
|
Randomized One-Group Posttest-Only Design |
|
|
237 | (2) |
|
Randomized Cross-Sectional and Longitudinal Survey Design |
|
|
239 | (2) |
|
One-Group Pretest-Posttest Design |
|
|
241 | (1) |
|
Comparison Group Posttest-Only Design |
|
|
241 | (1) |
|
Comparison Group Pretest-Posttest Design |
|
|
242 | (1) |
|
Interrupted Time-Series Design |
|
|
243 | (1) |
|
|
|
244 | (3) |
|
Classical Experimental Design |
|
|
245 | (1) |
|
Randomized Posttest-Only Control Group Design |
|
|
246 | (1) |
|
Summing Up and Looking Ahead |
|
|
247 | (1) |
|
|
|
247 | (3) |
|
|
|
250 | (2) |
|
References and Further Readings |
|
|
252 | (5) |
| PART III: ISSUES OF EVALUATION |
|
|
Politics, Ethics, and Standards |
|
|
257 | (28) |
|
|
|
257 | (1) |
|
Appropriate and Inappropriate Uses of Evaluation |
|
|
258 | (5) |
|
|
|
258 | (3) |
|
Proper Uses of Evaluation |
|
|
261 | (2) |
|
Political Influences on the Evaluation Process |
|
|
263 | (4) |
|
Manipulating the Evaluation Process |
|
|
263 | (1) |
|
Misdirecting the Evaluation Process |
|
|
264 | (3) |
|
Professional Standards for Evaluation |
|
|
267 | (8) |
|
|
|
268 | (1) |
|
|
|
269 | (1) |
|
|
|
269 | (6) |
|
|
|
275 | (1) |
|
|
|
275 | (1) |
|
Principles of Evaluation Practice |
|
|
276 | (3) |
|
Evaluation and Service Delivery Activities Should Be Integrated |
|
|
276 | (1) |
|
Involve from the Beginning as Many Stakeholder Groups as Possible |
|
|
277 | (1) |
|
Involve All Levels of Staff in the Evaluation Process |
|
|
278 | (1) |
|
Make Explicit the Purpose of the Evaluation |
|
|
278 | (1) |
|
Provide a Balanced Report and Disseminate Early and Regularly |
|
|
279 | (1) |
|
Summing Up and Looking Ahead |
|
|
279 | (1) |
|
|
|
280 | (1) |
|
|
|
280 | (2) |
|
References and Further Readings |
|
|
282 | (3) |
|
Culturally Appropriate Evaluations |
|
|
285 | (22) |
|
|
|
286 | (1) |
|
|
|
287 | (3) |
|
|
|
287 | (2) |
|
Intercultural Communications |
|
|
289 | (1) |
|
|
|
290 | (4) |
|
Orientation to Information |
|
|
291 | (1) |
|
|
|
291 | (1) |
|
|
|
292 | (1) |
|
|
|
292 | (1) |
|
|
|
293 | (1) |
|
|
|
294 | (6) |
|
|
|
294 | (1) |
|
Intercultural Communication Skills |
|
|
295 | (1) |
|
Developing Specific Knowledge About the Culture |
|
|
296 | (1) |
|
|
|
297 | (3) |
|
Summing Up and Looking Ahead |
|
|
300 | (1) |
|
|
|
301 | (1) |
|
|
|
301 | (1) |
|
References and Further Readings |
|
|
302 | (5) |
| PART IV: UTILITY OF EVALUATION |
|
|
Developing a Data Information System |
|
|
307 | (26) |
|
Staff Members' Roles in Developing a Data Information System |
|
|
308 | (1) |
|
Establishing an Organizational Plan |
|
|
309 | (4) |
|
Case-Level Data Collection |
|
|
310 | (3) |
|
Program-Level Data Collection |
|
|
313 | (1) |
|
Data Collection at Intake |
|
|
314 | (12) |
|
Data Collection at Client Contact |
|
|
318 | (4) |
|
Data Collection at Termination |
|
|
322 | (1) |
|
Data Collection to Obtain Feedback |
|
|
323 | (3) |
|
|
|
326 | (1) |
|
|
|
326 | (3) |
|
Computer-Assisted Data Management |
|
|
327 | (1) |
|
|
|
328 | (1) |
|
Summing Up and Looking Ahead |
|
|
329 | (1) |
|
|
|
330 | (1) |
|
|
|
330 | (1) |
|
References and Further Readings |
|
|
331 | (2) |
|
Decision Making with Objective and Subjective Data |
|
|
333 | (28) |
|
|
|
333 | (2) |
|
|
|
335 | (1) |
|
Case-Level Decision Making |
|
|
336 | (13) |
|
The Engagement and Problem-Definition Phase |
|
|
336 | (2) |
|
The Practice Objective--Setting Phase |
|
|
338 | (1) |
|
|
|
339 | (7) |
|
The Termination and Follow-Up Phase |
|
|
346 | (3) |
|
Program-Level Decision Making |
|
|
349 | (5) |
|
|
|
350 | (1) |
|
|
|
350 | (4) |
|
Using Outcome Monitoring Data In Program-Level Decision Making |
|
|
354 | (3) |
|
|
|
355 | (1) |
|
|
|
355 | (1) |
|
|
|
356 | (1) |
|
|
|
357 | (1) |
|
|
|
357 | (4) |
| Credits |
|
361 | (2) |
| Index |
|
363 | |