WEBVTT

00:00:00.266 --> 00:00:04.876
[Music] Welcome to the overview
on Evaluating Implementation

00:00:04.876 --> 00:00:07.906
and Performance to
Improve Program Quality.

00:00:10.366 --> 00:00:13.036
To ensure that expanded
learning time schools

00:00:13.036 --> 00:00:15.946
and out-of-school programs
are operating as designed

00:00:16.306 --> 00:00:19.386
and achieving their desired
goals, it is essential

00:00:19.386 --> 00:00:22.636
to routinely conduct
evaluations to assess quality

00:00:22.636 --> 00:00:24.846
of implementation
and student outcomes.

00:00:25.146 --> 00:00:29.246
Schools, districts, and program
providers can collaborate

00:00:29.416 --> 00:00:30.926
in collecting, analyzing,

00:00:30.926 --> 00:00:33.086
and interpreting the
evaluation results

00:00:33.086 --> 00:00:34.516
to inform decision making.

00:00:35.766 --> 00:00:38.036
Experts suggest using
both formative

00:00:38.036 --> 00:00:41.746
and summative evaluations
to monitor implementation

00:00:41.746 --> 00:00:43.466
and assess performance.

00:00:44.156 --> 00:00:47.166
Formative evaluations can
help determine which elements

00:00:47.166 --> 00:00:49.076
of implementation
are working well

00:00:49.076 --> 00:00:51.256
and where changes
need to be made.

00:00:51.866 --> 00:00:53.616
Summative evaluations are good

00:00:53.616 --> 00:00:55.976
for determining how
effective an intervention

00:00:56.146 --> 00:00:59.156
or a program has been in
accomplishing its goals,

00:00:59.336 --> 00:01:01.036
such as improving
student reading,

00:01:01.116 --> 00:01:02.886
writing, or math performance.

00:01:03.556 --> 00:01:06.006
A rigorous program
evaluation involves:

00:01:06.346 --> 00:01:10.236
Developing an evaluation
plan,   Collecting program

00:01:10.236 --> 00:01:13.656
and student performance
data,   Analyzing the data

00:01:13.656 --> 00:01:17.766
to determine program impact and
identify areas for improvement,

00:01:17.906 --> 00:01:21.686
Sharing evaluation
findings with program providers,

00:01:22.046 --> 00:01:24.986
and   Taking steps
to ensure the quality

00:01:24.986 --> 00:01:26.506
of the evaluation itself.

00:01:27.576 --> 00:01:31.166
Developing an evaluation plan
is a critical initial step

00:01:31.716 --> 00:01:34.876
to help guide the
evaluation process and ensure

00:01:34.876 --> 00:01:37.646
that it produces accurate
and informative results.

00:01:38.446 --> 00:01:40.946
All stakeholders,
including teachers, parents,

00:01:40.946 --> 00:01:43.866
school and district leaders,
and program administrators

00:01:43.866 --> 00:01:46.646
and staff, should be
involved in this process.

00:01:47.086 --> 00:01:48.946
An evaluation plan should have:

00:01:48.946 --> 00:01:52.856
Measurable goals,  
Clear research questions,

00:01:52.856 --> 00:01:57.056
A data collection plan,
A data analysis plan,

00:01:57.056 --> 00:02:01.216
A dissemination plan, and 
A timeline.

00:02:01.276 --> 00:02:04.776
Schools and districts
need to develop, adapt,

00:02:04.796 --> 00:02:08.276
or select reliable instruments
for measuring fidelity

00:02:08.576 --> 00:02:10.916
of implementation,
student outcomes,

00:02:10.936 --> 00:02:13.716
and stakeholder satisfaction.

00:02:13.746 --> 00:02:16.046
Program implementation
data can be gathered

00:02:16.046 --> 00:02:19.466
on curriculum content,
instructional delivery,

00:02:19.526 --> 00:02:22.236
staff-student interaction,
and behavior

00:02:22.236 --> 00:02:25.996
of staff members including
instruction in the classroom,

00:02:26.076 --> 00:02:29.646
teacher collaboration,
and planning practices.

00:02:30.716 --> 00:02:33.486
Observation checklists can be used
for collecting program monitoring data.

00:02:34.666 --> 00:02:39.206
Out-of-school providers and schools 
can share information collected locally.

00:02:39.206 --> 00:02:43.376
For example, out-of-school
program staff can provide data

00:02:43.376 --> 00:02:46.066
on student attendance and
behavior while they are

00:02:46.066 --> 00:02:48.306
in the program as
well as the results

00:02:48.306 --> 00:02:51.966
of any academic assessments
conducted by program instructors.

00:02:52.446 --> 00:02:55.246
Schools and districts
can compile course grades

00:02:55.246 --> 00:02:57.466
or provide access
to data systems.

00:02:58.146 --> 00:03:00.146
They can also provide
the results of state

00:03:00.146 --> 00:03:02.046
and district standardized tests.

00:03:03.556 --> 00:03:06.376
Additionally, key stakeholders,
including principals,

00:03:06.376 --> 00:03:08.826
classroom teachers,
parents, and students,

00:03:09.256 --> 00:03:10.876
can provide important
information

00:03:10.876 --> 00:03:13.546
on how well the program is
meeting everyone's needs.

00:03:14.086 --> 00:03:16.136
This information can
be gathered by phone,

00:03:16.436 --> 00:03:18.246
focus groups, or surveys.

00:03:19.256 --> 00:03:22.476
While collaborating in data
collection, schools, districts,

00:03:22.476 --> 00:03:25.616
and program providers can
coordinate their activities

00:03:25.886 --> 00:03:28.276
to eliminate duplicate
data collection efforts.

00:03:29.386 --> 00:03:32.176
Schools and districts can
carefully analyze the data

00:03:32.176 --> 00:03:36.096
collected and use the findings
to improve program quality.

00:03:36.626 --> 00:03:40.856
Analysis strategies should:  
Identify any inconsistencies

00:03:40.856 --> 00:03:43.566
between what the increased
learning time model was designed

00:03:43.566 --> 00:03:47.896
to do and how it is actually
operating,   Determine

00:03:47.896 --> 00:03:51.326
which aspects of the model are
working well and which are not,

00:03:51.566 --> 00:03:54.586
Compare the model
outcomes with outcomes

00:03:54.586 --> 00:03:57.466
of other models operating
in comparable schools

00:03:57.506 --> 00:04:00.896
or with comparable student
populations, and   Look

00:04:00.896 --> 00:04:04.106
for performance patterns in
the data to identify subgroups

00:04:04.106 --> 00:04:06.696
of students that benefit
from the model the most,

00:04:07.186 --> 00:04:10.186
as well as any contextual
factors that facilitated

00:04:10.336 --> 00:04:11.996
or impeded implementation.

00:04:13.226 --> 00:04:16.486
Sharing evaluation findings
and conducting discussions

00:04:16.486 --> 00:04:17.685
about these findings

00:04:17.685 --> 00:04:19.995
with expanded learning
day school staff

00:04:19.995 --> 00:04:23.096
or out-of-school program
instructors can greatly help

00:04:23.096 --> 00:04:25.236
to improve the quality
of those models.

00:04:26.046 --> 00:04:29.086
Through discussions around
data, districts, schools,

00:04:29.086 --> 00:04:32.376
and program providers may
decide what additional support

00:04:32.376 --> 00:04:35.616
for the program is needed, such
as professional development

00:04:36.036 --> 00:04:39.246
or helping the program better
align its curriculum with that

00:04:39.246 --> 00:04:40.586
of the regular classroom.

00:04:40.716 --> 00:04:44.846
If the data consistently show
that students are not improving,

00:04:45.426 --> 00:04:46.736
districts, schools,

00:04:46.796 --> 00:04:49.536
and providers might consider
choosing a different program

00:04:49.536 --> 00:04:51.916
model or ending the
current program.

00:04:53.106 --> 00:04:56.686
Rigorous independent evaluations
provide the best measurements

00:04:56.686 --> 00:04:59.306
of a program's ability to
raise student achievement.

00:04:59.716 --> 00:05:03.456
Positive findings from such
evaluations can greatly boost

00:05:03.456 --> 00:05:06.996
both participation in and
funding for the program.

00:05:07.986 --> 00:05:11.486
By hiring an external evaluator
to conduct the evaluation,

00:05:11.956 --> 00:05:14.926
schools and districts can
help ensure the credibility

00:05:14.926 --> 00:05:17.286
of the evaluation
and its findings.

00:05:18.156 --> 00:05:19.486
If this is not an option,

00:05:19.676 --> 00:05:22.656
experienced evaluators can
be consulted about the plans

00:05:22.656 --> 00:05:24.156
and methods being considered.

00:05:25.366 --> 00:05:28.606
If access to experienced
external evaluators is not

00:05:28.606 --> 00:05:32.946
possible, schools and districts
can consider using existing

00:05:32.946 --> 00:05:37.536
staff members who have expertise
in evaluation and data analysis.

00:05:38.046 --> 00:05:41.346
These in-house experts may
also help program providers

00:05:41.346 --> 00:05:44.546
understand evaluation
findings and use these data

00:05:44.546 --> 00:05:46.896
to develop program
improvement plans.

00:05:47.496 --> 00:05:49.316
Local colleges and universities

00:05:49.476 --> 00:05:53.226
or parent teacher associations
may also provide data collection

00:05:53.226 --> 00:05:54.716
and analysis services.

00:05:55.236 --> 00:05:57.006
In addition, the
federal government,

00:05:57.006 --> 00:05:58.836
state education agencies,

00:05:59.156 --> 00:06:03.066
and private foundations may
help identify funding sources

00:06:03.066 --> 00:06:04.666
to support the evaluation.

00:06:05.796 --> 00:06:08.286
Districts play a
vital role in ensuring

00:06:08.286 --> 00:06:11.986
that extended learning time
programs effectively advance

00:06:11.986 --> 00:06:12.966
student achievement.

00:06:13.426 --> 00:06:15.066
By initiating, supporting,

00:06:15.196 --> 00:06:17.736
and facilitating program
evaluation efforts,

00:06:18.276 --> 00:06:21.696
districts are well informed on
whether specific interventions

00:06:21.696 --> 00:06:25.466
or practices need to
be revised, replaced,

00:06:25.466 --> 00:06:26.976
or replicated at
additional sites.

00:06:28.466 --> 00:06:31.686
To learn more about Evaluating
Implementation and Performance

00:06:31.686 --> 00:06:35.946
to Improve Program Quality,
explore the additional resources

00:06:35.946 --> 00:06:37.966
on the Doing What Works website.

00:06:37.966 --> 00:06:38.000
[Music]