Context
Organization A is a volunteer literacy organization that provides free literacy services to individuals, families, workplaces and the community.
Their vision: To support create a community that values literacy.
Their mission: To provide literacy services for adults and families.
This non-profit organization offers many programs to adults through 4 focus areas:
1. One-to-one tutoring program
2. Family literacy programs and services
3. Facilitator trainings
4. Workplace programs
The management is undertaking the development of an Accountability Framework to provide a roadmap for their agency (for all 4 focus areas offered), to meet the needs of major funders and to share the framework with four other agencies once the framework is completed.
Stakeholders Involved
The audience for the information from the evaluation is management and staff in the initial phase and funders and 4 other organizations at a later stage. The approach taken to the evaluation would be participatory, especially when examining outcomes and measures, not only to ensure the information is relevant and that commitment would likely be stronger, but also to enable staff to conduct and use their own evaluation that is realistic, relevant and doable in their practice after the evaluation is completed.
Purpose of Evaluation
The purpose is to improve the measurement of program outcomes in order to enhance and verify the impact of Literacy programs/services in order to:
· improve program effectiveness
· guide future program development
· prepare and justify budgets
· focus board members’ attention to program issues and values
· improve and promote clarity and collaboration with funders by employing a common set of terms and language
· show funders that benefits produced merit continued support
· prove that informal education is not only valuable, but also valid and measureable
· reinforce the positive impact staff and volunteers make as a result of their dedication and hard work.
Type of Evaluation
The evaluation would be outcomes-based. It is a long term project (2 years) that will require an examination and identification of the current outcomes for the 4 main focus areas and the services and programs found within each area. Identification, enhancement &/or development of measures and efficient methods to collect information would follow. It is important to note that many participants have low literacy levels, so creative and non-print based techniques to collect data will be required.
Within the One-to-one tutoring focus area (and possibly the Family Literacy focus area), Adult Literacy Benchmarks will be integrated with the existing program outcomes and measures will need to be developed to address them. This will require close collaboration and planning with management and coordinators throughout all phases of the project. Since this project has yet to begin, these are preliminary plans that will change as more information and details develop.
Data Collection
· Program documents (program evaluations, anecdotal records)
· Funding proposals for programs (& other documents that list inputs, outputs, activities and outcomes for Organization A)
· Past and current reports, both external (to funders and board members) and internal (staff monthly reports)
· Staff and management interviews and/or surveys
· Observing program participants (re: the program evaluations)
· Conducting focus groups
Thursday, October 15, 2009
Saturday, September 19, 2009
Assignment 2: ECUR 809
Suggested Evaluation Model Re:ECS Program for Children with Severe Disabilities
Some key factors make this program more suitable to some evaluation models than others:
1. The program is customized and in a sense, new every time in regards to goals and objectives. Specific goals-based evaluation would be on a case-by-case basis. However, the general process could be examined and compared (timelines, reasons for adapting goals and/or barriers to achieving goals).
2. The unique and individualized nature of the content (individual children’s needs direct the choice of activities) make it challenging to attempt generalized comparisons.
3. The variation in program contexts: centre-based and in-home and possibly a combination of both add to the difficulty of using a broad brush to paint the overall program results.
4. The various and changing stakeholders (children, parents, teachers, and/or teacher assistants, speech/language pathologist, and funder: Alberta Education) make systematic comparisons difficult at best.
Depending on resources available (time and money), I’d approach the evaluation using parts of Provus’ Discrepancy Model (DIPPS) to compare and contrast the two contexts of home (one-to-one) and classroom (group). Since Design content is specific and Installation appears established with the criteria regulated quite firmly, I’d focus on the Process, Product and Cost-benefit analysis.
Some Possible Methods to collect data: observation and program visits, interviews with staff and parents, case studies involving in-home and center-based program participants (as feasible and appropriate), key stakeholder surveys (pre- and post- program), participant assessment results (pre and post program), a cost-benefit analysis to include a review documentation to assess budget, other similar existing programs, and program schedules.
I would classify this as a participatory or collaborative approach, with all stakeholders contributing advice, expertise, and rich perspectives in order to best serve the needs of the children who benefit from this program.
Resource: Medicine Hat Catholic Separate Regional Division No. 20, Student Services. 2009.
Student Services Program Description, ECS Programming for Children with Severe Disabilities. www.mhcbe.ab.ca
Some key factors make this program more suitable to some evaluation models than others:
1. The program is customized and in a sense, new every time in regards to goals and objectives. Specific goals-based evaluation would be on a case-by-case basis. However, the general process could be examined and compared (timelines, reasons for adapting goals and/or barriers to achieving goals).
2. The unique and individualized nature of the content (individual children’s needs direct the choice of activities) make it challenging to attempt generalized comparisons.
3. The variation in program contexts: centre-based and in-home and possibly a combination of both add to the difficulty of using a broad brush to paint the overall program results.
4. The various and changing stakeholders (children, parents, teachers, and/or teacher assistants, speech/language pathologist, and funder: Alberta Education) make systematic comparisons difficult at best.
Depending on resources available (time and money), I’d approach the evaluation using parts of Provus’ Discrepancy Model (DIPPS) to compare and contrast the two contexts of home (one-to-one) and classroom (group). Since Design content is specific and Installation appears established with the criteria regulated quite firmly, I’d focus on the Process, Product and Cost-benefit analysis.
Some Possible Methods to collect data: observation and program visits, interviews with staff and parents, case studies involving in-home and center-based program participants (as feasible and appropriate), key stakeholder surveys (pre- and post- program), participant assessment results (pre and post program), a cost-benefit analysis to include a review documentation to assess budget, other similar existing programs, and program schedules.
I would classify this as a participatory or collaborative approach, with all stakeholders contributing advice, expertise, and rich perspectives in order to best serve the needs of the children who benefit from this program.
Resource: Medicine Hat Catholic Separate Regional Division No. 20, Student Services. 2009.
Student Services Program Description, ECS Programming for Children with Severe Disabilities. www.mhcbe.ab.ca
Friday, September 11, 2009
Assignment 1 ECUR 809
Assignment 1: ECUR 809
September 11, 2009
For: Jay Wilson
By: Joanne LaBrash
Lessons Learned from a State-Funded Workplace Literacy Program
Project Summary/Background
An evaluation was completed on a set of ten workplace literacy pilot programs that were publicly funded by a state government (The Indiana Department of Workforce Development). One of the identified purposes for the projects came from: “the perception that Indiana’s competitiveness and growth may be constrained by a mature, incumbent workforce that has deficient basic skills” (p 2). These pilot programs were developed to improve those perceived deficient skills.
This evaluation is both formative and summative: formative because it examines a pilot project; (its future improvements, adaptations, or termination depend on the evaluation results), and summative because it quantified scores to measure learner gains. (1800 participant surveys and test scores were collected and analyzed).
The model showed elements of Provus’ Discrepancy Model, in that it compared intentions with actual results in the areas of DIPPS: design and content, installation, process, product and cost-benefit analysis (to varying degrees). Both qualitative and quantitative data were used and I think that the combination of methods adds valuable layers of perspective.
Qualitative Focus
Two site visits occurred at ‘most’ of the 10 funded projects. I would have liked all 10 sites visited, which would give a more thorough analysis and evaluation. The first visit focused on planning and implementation (design, installation and process) and the second visit near the end of the project focused on outcomes and lessons learned (product and cost benefit analysis). The visits involved examination of administrative challenges, instructional delivery, contextualization, motivations of both employee participants and businesses, and a general cost-benefit analysis for workers, companies, literacy providers, and government.
The struggle with implementation was highlighted. None of the sites were familiar with the pre and post test requirements, which led to strained relationships with workers and employers when it was determined that a third assessment appraisal was needed. This confusion led to inconsistency across all sites, with only three sites administering the pre-assessment test and only one-third completing post-tests. The results showed higher scores for pre-tests than post-tests overall! I tended to discount the validity and use of the data for these reasons.
Another implementation failure was that the computer course required internet access, which was not always available. The resource materials also revealed problems (especially the computer course) with “pushback from students (and employers) who felt it was too technical” (p 4).
Quantitative Focus
A survey of participants, the adult student assessment test scores and resulting learning gains along with earnings histories were the data quantitatively analyzed. I noticed a failure to draw any strong connection between the earnings histories to the impact desired; the state hoped to reduce the use of public assistance benefits. The inclusion of possible additional factors involved and a longer time frame were neglected. Therefore, the information gathered seemed to be irrelevant and a bit misguided, although it probably looked good in a graph. Statistics tend to hold more appeal and appear objective, but for what purpose?
Overall, I found the evaluation very effective, with strong examination of design, content, installation and process. Its weakest area was the evaluation of the product. In my view, the attempts to measure outcomes and changes were riddled with testing and resource material problems that negatively impacted the validity intended. All of the qualitative data and most of the quantitative data were directly connected to the findings and were purposefully and thoughtfully examined.
Reference:
Hollenbeck, Kevin and Timmeney, Bridget. (2009). Lessons Learned from a State-Funded
Workplace Literacy Program W.E. Upjohn Institute for Employment Research. Indiana.
(Working Paper 09-146). http://ssrn.com/abstract=1359182
September 11, 2009
For: Jay Wilson
By: Joanne LaBrash
Lessons Learned from a State-Funded Workplace Literacy Program
Project Summary/Background
An evaluation was completed on a set of ten workplace literacy pilot programs that were publicly funded by a state government (The Indiana Department of Workforce Development). One of the identified purposes for the projects came from: “the perception that Indiana’s competitiveness and growth may be constrained by a mature, incumbent workforce that has deficient basic skills” (p 2). These pilot programs were developed to improve those perceived deficient skills.
This evaluation is both formative and summative: formative because it examines a pilot project; (its future improvements, adaptations, or termination depend on the evaluation results), and summative because it quantified scores to measure learner gains. (1800 participant surveys and test scores were collected and analyzed).
The model showed elements of Provus’ Discrepancy Model, in that it compared intentions with actual results in the areas of DIPPS: design and content, installation, process, product and cost-benefit analysis (to varying degrees). Both qualitative and quantitative data were used and I think that the combination of methods adds valuable layers of perspective.
Qualitative Focus
Two site visits occurred at ‘most’ of the 10 funded projects. I would have liked all 10 sites visited, which would give a more thorough analysis and evaluation. The first visit focused on planning and implementation (design, installation and process) and the second visit near the end of the project focused on outcomes and lessons learned (product and cost benefit analysis). The visits involved examination of administrative challenges, instructional delivery, contextualization, motivations of both employee participants and businesses, and a general cost-benefit analysis for workers, companies, literacy providers, and government.
The struggle with implementation was highlighted. None of the sites were familiar with the pre and post test requirements, which led to strained relationships with workers and employers when it was determined that a third assessment appraisal was needed. This confusion led to inconsistency across all sites, with only three sites administering the pre-assessment test and only one-third completing post-tests. The results showed higher scores for pre-tests than post-tests overall! I tended to discount the validity and use of the data for these reasons.
Another implementation failure was that the computer course required internet access, which was not always available. The resource materials also revealed problems (especially the computer course) with “pushback from students (and employers) who felt it was too technical” (p 4).
Quantitative Focus
A survey of participants, the adult student assessment test scores and resulting learning gains along with earnings histories were the data quantitatively analyzed. I noticed a failure to draw any strong connection between the earnings histories to the impact desired; the state hoped to reduce the use of public assistance benefits. The inclusion of possible additional factors involved and a longer time frame were neglected. Therefore, the information gathered seemed to be irrelevant and a bit misguided, although it probably looked good in a graph. Statistics tend to hold more appeal and appear objective, but for what purpose?
Overall, I found the evaluation very effective, with strong examination of design, content, installation and process. Its weakest area was the evaluation of the product. In my view, the attempts to measure outcomes and changes were riddled with testing and resource material problems that negatively impacted the validity intended. All of the qualitative data and most of the quantitative data were directly connected to the findings and were purposefully and thoughtfully examined.
Reference:
Hollenbeck, Kevin and Timmeney, Bridget. (2009). Lessons Learned from a State-Funded
Workplace Literacy Program W.E. Upjohn Institute for Employment Research. Indiana.
(Working Paper 09-146). http://ssrn.com/abstract=1359182
Subscribe to:
Posts (Atom)