What is Program Evaluation?
Program evaluation is the systematic investigation of a project or program to determine its worth or merit. Rigorous evaluation is a complex enterprise that is:
- designed for a specific purpose
- conducted for action-related reasons
The challenge for evaluators is to use valid models, strategies, and tools that will help educators in schools, districts, and state education agencies determine the extent to which programs make a difference in teaching, learning, and the school environment. An evaluation is useful if it yields information to support decisions about prudent courses of action.
Why Evaluate Programs?
When program administrators have data pertaining to the implementation of their program activities, they can identify needs, problems, and opportunities, and make informed decisions about which aspects of the program should be sustained, improved, or eliminated.
When policymakers and other stakeholders, such as school boards or funding agencies, have data indicating that a program is meeting its goals and objectives, which usually means improving teaching and learning, they are more willing to support program continuation.
How to Evaluate Technology Programs?
In this era of accountability, technology Plans for schools, districts, and states must not only spell out what the program will do and what resources will be installed, but also the evaluation strategies that will provide program impact data.
Schools, districts, and states must show a connection between program activities and student outcomes. This means that along with goals, objectives, and activities, the technology plan must include indicators, benchmarks, and data sources.
Challenges to Technology Program Evaluation
- All too often, technology plans lack a component for evaluating the success and effectiveness of the program. The plan may designate a person or committee with oversight for reporting technology progress, but little else. This frequently means that there is little or no in-house expertise in conducting an evaluation of effective technology integration that will yield meaningful and useful results.
- Standardized tests seldom measure the variables that technology most like enhances, like creativity, problem-solving, critical thinking, design, school attendance rate, dropout rate, and discipline referrals. However, many political and community leaders are requiring the documented impact of technology use on test scores.
- Designing an evaluation plan requires substantial levels of specialized training.
- The rule of thumb regarding the cost of program evaluation is this: at least ten percent of a project budget should be spent on evaluation. Many program administrators plan to spend program money on staff, infrastructure, or professional development, not evaluation.
- Not many educators have expertise in both technology integration and program evaluation, so it is difficult to find a good evaluator.
Guidelines for Program Evaluation
Research and best practices indicate that a typical process for evaluation of technology efforts involves giving attention to the following steps:
- Identify basic issues/problems that need attention
- Determine data necessary for making informed decisions
- Determine the usefulness of existing data
- Determine new data that must be collected
- Identify evaluation design
- Determine tools and data acquisition strategies
- Determine timelines and assign staff roles
- Conduct evaluations
- Analyze data and develop conclusions
- Report data to address various audience(s) needs
Unfortunately, few districts or schools have the capacity or resources for designing and conducting rigorous program evaluations. Many seek external evaluators to design and conduct their technology program evaluations.
Joint Committee on Standards for Educational Evaluation (1994)