WEBVTT

00:00:00.516 --> 00:00:04.546
[Music]

00:00:05.046 --> 00:00:06.876
My name is Danette Parsley.

00:00:06.876 --> 00:00:09.336
I am a senior director for
field services at McREL,

00:00:09.406 --> 00:00:11.086
Mid-continent Research
for Education

00:00:11.086 --> 00:00:12.526
and Learning, in
Denver, Colorado.

00:00:13.546 --> 00:00:15.766
When it comes to evaluating
programs effectively,

00:00:15.766 --> 00:00:17.556
there are a few key
ideas to keep in mind.

00:00:18.046 --> 00:00:22.216
One is that they should be
intentional and well planned.

00:00:22.216 --> 00:00:26.016
Another idea is that they should
involve multiple stakeholders

00:00:26.016 --> 00:00:27.986
at the various phases
of evaluation.

00:00:28.636 --> 00:00:33.716
And finally, evaluation should
really be an integrated ongoing

00:00:33.716 --> 00:00:37.516
part of your program and not a
stand-alone or one-time event.

00:00:38.526 --> 00:00:41.186
When you begin planning your
evaluation, it's important

00:00:41.186 --> 00:00:44.196
to involve multiple
stakeholders in the process.

00:00:44.606 --> 00:00:47.316
You want to involve key
people who really have a stake

00:00:47.706 --> 00:00:49.346
in the outcome of your program.

00:00:49.526 --> 00:00:53.596
So you want to involve parents,
community members, teachers,

00:00:53.696 --> 00:00:57.786
principals, and also
instructors and administrators

00:00:57.786 --> 00:00:58.976
for your actual program.

00:00:59.346 --> 00:01:00.826
You might also include people

00:01:00.826 --> 00:01:02.986
like partners or
funders as well.

00:01:03.376 --> 00:01:05.926
It depends on the goals and the
outcomes that you are trying

00:01:05.926 --> 00:01:07.626
to obtain for your program.

00:01:08.346 --> 00:01:10.806
Now, one of the first things
that you often want to do is lay

00:01:10.806 --> 00:01:14.776
out logically your program
in sort of a snapshot

00:01:14.816 --> 00:01:18.666
by articulating a theory
of change or a logic model.

00:01:19.016 --> 00:01:22.696
A logic model helps you
articulate the key components

00:01:22.696 --> 00:01:26.456
or elements of your program and
your rationale for your program.

00:01:26.866 --> 00:01:32.386
So it often contains
elements such as your vision

00:01:32.466 --> 00:01:37.206
for your program, your context,
so what are your driving forces

00:01:37.206 --> 00:01:42.056
for having your program, such
as, we have a lot of kids

00:01:42.056 --> 00:01:44.736
who are not attending
school regularly,

00:01:45.096 --> 00:01:48.246
we have an achievement
gap between subgroups

00:01:48.246 --> 00:01:52.496
of our students, we have a high
dropout rate perhaps-things

00:01:52.496 --> 00:01:54.086
like that.

00:01:54.086 --> 00:01:57.126
So a little bit of contextual
information as far as the reason

00:01:57.126 --> 00:01:58.506
for your program existing.

00:01:59.266 --> 00:02:01.536
The next thing you want to
do is lay out your short-

00:02:01.536 --> 00:02:03.756
and long-term outcomes
for your program.

00:02:03.756 --> 00:02:06.086
So what are the results that
you want for your program?

00:02:06.126 --> 00:02:07.556
So what are you trying
to accomplish

00:02:07.556 --> 00:02:08.756
by having this program exist?

00:02:10.006 --> 00:02:12.696
Finally, you really want
to lay out your strategies

00:02:12.696 --> 00:02:15.576
and activities that
you are going to use

00:02:15.606 --> 00:02:16.786
to try to get those results.

00:02:17.646 --> 00:02:21.166
From there, your evaluation
plan really falls into place.

00:02:21.446 --> 00:02:23.716
So once you have that
snapshot of your program

00:02:23.716 --> 00:02:26.236
and you really agree
to what that is

00:02:26.236 --> 00:02:27.336
with the key stakeholders,

00:02:27.606 --> 00:02:31.056
then your evaluation plan
falls naturally from there.

00:02:31.056 --> 00:02:33.716
A sound evaluation plan has
several characteristics.

00:02:34.486 --> 00:02:37.516
First, you want to articulate
your purpose and the type

00:02:37.516 --> 00:02:39.136
of evaluation that you
are going to conduct.

00:02:39.796 --> 00:02:43.426
And by type of evaluation, I
mean formative or summative,

00:02:44.006 --> 00:02:45.646
and you may want to do both.

00:02:46.036 --> 00:02:48.536
So your formative
evaluation would be more

00:02:48.536 --> 00:02:50.176
of a process evaluation,

00:02:51.146 --> 00:02:55.016
and that would be assessing
implementation of your program

00:02:55.516 --> 00:03:01.416
and evaluating the progress
toward meeting your outcomes.

00:03:02.156 --> 00:03:05.576
The summative evaluation would
be about ultimate impacts

00:03:05.576 --> 00:03:08.436
or your outcomes; it is more
of an outcome evaluation.

00:03:08.966 --> 00:03:11.956
The next thing that you
would want to articulate,

00:03:11.956 --> 00:03:14.956
and this is closely aligned
with your purpose and type

00:03:14.956 --> 00:03:17.906
of evaluation, would
be your questions,

00:03:18.126 --> 00:03:19.126
your evaluation questions.

00:03:19.726 --> 00:03:24.716
So you may want to
ask a question like

00:03:24.716 --> 00:03:28.616
"What is the impact of our
program for our students?"

00:03:29.076 --> 00:03:31.746
Or you may say, "What is
the impact of our program

00:03:31.746 --> 00:03:34.836
for particular subgroups
of students?"

00:03:34.836 --> 00:03:38.626
You may want to ask a more
formative question such as

00:03:38.696 --> 00:03:43.216
"Are we progressing toward
our targets or our goals?"

00:03:43.216 --> 00:03:46.786
Or you may have a more
descriptive question such as

00:03:46.786 --> 00:03:48.336
"What services are
we providing?"

00:03:48.796 --> 00:03:52.506
or "Are we implementing
our program as intended?"

00:03:52.946 --> 00:03:55.706
So those are different types
of evaluation questions.

00:03:56.326 --> 00:03:59.776
It's important to think about
roles and responsibilities

00:03:59.776 --> 00:04:01.926
as far as data collection
efforts,

00:04:02.546 --> 00:04:04.676
and that's often a
shared responsibility

00:04:04.676 --> 00:04:07.746
between program staff and
regular day school staff.

00:04:08.456 --> 00:04:12.076
So, for example, existing
data, like achievement data

00:04:12.076 --> 00:04:15.626
on standardized tests or
benchmark assessments or grades,

00:04:15.996 --> 00:04:18.486
that may be collected
by a school staff member

00:04:18.846 --> 00:04:20.336
and provided to the program.

00:04:21.245 --> 00:04:25.406
Other data, such as
stakeholder satisfaction data

00:04:26.016 --> 00:04:31.616
or program implementation data,
that may be collected either

00:04:31.876 --> 00:04:35.876
as a joint effort between
the program and the school

00:04:35.876 --> 00:04:37.116
or by the program itself.

00:04:37.116 --> 00:04:39.996
Often schools have
school improvement teams

00:04:39.996 --> 00:04:44.186
or leadership teams who are
responsible for analyzing data

00:04:44.876 --> 00:04:46.376
to inform their school
improvement plan,

00:04:46.376 --> 00:04:50.056
so they set goals and
articulate strategies

00:04:50.096 --> 00:04:50.996
for making improvements.

00:04:51.456 --> 00:04:54.486
It's important for programs
to work collaboratively

00:04:54.796 --> 00:04:58.096
with these teams, and they
will help build the capacity

00:04:58.426 --> 00:04:59.976
of the providers in using data.

00:05:01.516 --> 00:05:07.500
[Music]