Grade 4 Reading Reinforcement Club
Program Context:
I am going to focus my evaluation on a grade four reading reinforcement club. The school in which this program take place is a K-8 private school located in Oakville, Ontario. This school is known in the community for offering additional one-on-one or small group programming for students who need reinforcement or enrichment of grade level work both during class time as well as before and after school. Parents can pay an additional fee to have individualized lessons for students which take place in the Learning Success Centre (LSC) taught by LSC teachers. Students may be exempted from certain subject areas (such as French) and attend the LSC during these periods. Students may also attend the LSC during other subject times such as science and learn the same lesson in a one on one setting. Many of these students also participate in additional academic improvement programs which are offered free of charge such as reading reinforcement club, peer mentoring, math contest club, or study skill sessions. These programs can take place during academic periods, lunch or after school I am going to focus my evaluation on the reading reinforcement club and will specifically focus on the 4th grade club. Program Focus: This Reading Reinforcement program is focussed on improving the reading abilities of students who are currently reading below grade level in grade four. Classroom teachers evaluate students at the beginning of the year using the Fountas and Pinnell Benchmark Assessment System (BAS). They note which students are currently reading below a level P which is the school standard for beginning of 4th grade. Please see the Reading Levels chart to the right of this post for more information on reading levels. Classroom teachers submit the names of the students to the Learning Success Centre (LSC) coordinator who then arranges groups of students based on reading ability. Students meet in small groups (2-5 students) with a LSC teacher twice a week to read a teacher-chosen text. They are pulled out of regular class time and meet for approximately 20 minutes per session. Students are provided explicit small-group instruction to increase their literacy skills; fluency, decoding and comprehension. |
Program History:
The program began in 2012 for grades 1-3 and was expanded to include grades 4-6 in 2013. The impetus to begin the program was the school's movement away from a whole class novel study into a balanced literacy approach. With this new approach, more emphasis was put on individual reading levels and the skills associated (decoding, fluency, comprehension).
Size of Staff:
The staff of this program consists of one LSC teacher who conducts the program at the grade four level. She is accountable to the LSC coordinator. The Reading Recovery teacher also coordinates with classroom teachers of the individual students.
Community Demographics
The school population is generally affluent. Parents pay approximately $17,000 Canadian per year for their students to attend this school. In addition, if their child requires LSC sessions, they pay an additional fee per session. Parents have high expectations for success in all areas. There is a large expatriate population at this school. Many students have attended 2 or more schools in their careers and often have attended schools overseas.
Program Goals:
The goal of the program is to provide extra reading support for students who are reading below grade level and to improve their reading ability to meet grade level expectations. The program is designed to provide direct teaching in a small group setting in the areas of fluency, decoding and comprehension.
Resources:
The resources include a secluded space in the school with limited distractions. This can take place in the LSC, or in any free classrooms. The program requires one teacher per group. The teacher requires levelled text readers for each child. Assessments are required to track student progress.
Purpose of Evaluation:
I am evaluating this program to evaluate the effectiveness of the program. Students are not being evaluated once in the program to see if they are in-fact improving their reading abilities.
Original Guiding Questions (See Theory of Change Section for Key Evaluation Questions)
The program began in 2012 for grades 1-3 and was expanded to include grades 4-6 in 2013. The impetus to begin the program was the school's movement away from a whole class novel study into a balanced literacy approach. With this new approach, more emphasis was put on individual reading levels and the skills associated (decoding, fluency, comprehension).
Size of Staff:
The staff of this program consists of one LSC teacher who conducts the program at the grade four level. She is accountable to the LSC coordinator. The Reading Recovery teacher also coordinates with classroom teachers of the individual students.
Community Demographics
The school population is generally affluent. Parents pay approximately $17,000 Canadian per year for their students to attend this school. In addition, if their child requires LSC sessions, they pay an additional fee per session. Parents have high expectations for success in all areas. There is a large expatriate population at this school. Many students have attended 2 or more schools in their careers and often have attended schools overseas.
Program Goals:
The goal of the program is to provide extra reading support for students who are reading below grade level and to improve their reading ability to meet grade level expectations. The program is designed to provide direct teaching in a small group setting in the areas of fluency, decoding and comprehension.
Resources:
The resources include a secluded space in the school with limited distractions. This can take place in the LSC, or in any free classrooms. The program requires one teacher per group. The teacher requires levelled text readers for each child. Assessments are required to track student progress.
Purpose of Evaluation:
I am evaluating this program to evaluate the effectiveness of the program. Students are not being evaluated once in the program to see if they are in-fact improving their reading abilities.
Original Guiding Questions (See Theory of Change Section for Key Evaluation Questions)
- How are LSC teachers being trained to run this program?
- What resources are LSC teachers using? Are these resources being used effectively?
- What data needs to be collected to assess the program effectiveness on an ongoing basis?
- How will we use the data to assess the program?
- What factors outside the program could be affecting its delivery and effectiveness?
- Are students who are attending Reading Recovery improving their reading ability based on the Benchmark Assessment System? How quickly? By how many levels?
- Were students engaged in the program? How is this engagement measured?
- What is being communicated with classroom teachers and parents of participants and how is this being communicated?
Theory of Change
If the Reading Reinforcement is implemented then the reading scores of the targeted students will increase. The focus of the program is to provide extra reading support for students who are reading below grade level and to improve their reading ability to meet grade level expectations. The program is designed to provide direct teaching in a small group setting in the areas of fluency, decoding and comprehension. |
Action Model
Change Model
Logic Model
Key Evaluation Questions
|
Evaluation Approach- Impact Evaluation
The reading reinforcement program will be evaluated based on the impact evaluation approach. This approach was chosen because the program is already up and running and is already producing results. The Better Evaluation Website states that "An impact evaluation provides information about the impacts produced by an intervention." In this case the Reading Reinforcement Program is the intervention. Since the Reading Reinforcement program happens each year with new students, "the findings of an impact evaluation can be used to improve implementation of a programme for the next intake of participants by identifying critical elements to monitor and tightly manage." (Better Evaluation Website).
This approach allows evaluators and the program stakeholders to see if the program is reaching its targeted outcomes. This type of evaluation provides valuable feedback for the stakeholders. The evaluation will:
This approach allows evaluators and the program stakeholders to see if the program is reaching its targeted outcomes. This type of evaluation provides valuable feedback for the stakeholders. The evaluation will:
- identify ways to measure the expected outcomes using concrete data
- collect and measure the data
- evaluate based on the collected data
- make recommendations to improve the effectiveness of the program and to limit the discrepancies between the actual and targeted outcomes.
Data Collection and Analysis
The Better Evaluation website states: Impact evaluations need to go beyond assessing the size of the effects (i.e., the average impact) to identify for whom and in what ways a programme or policy has been successful. What constitutes ‘success’ and how the data will be analysed and synthesized to answer the specific key evaluation questions (KEQs) must be considered up front as data collection should be geared towards the mix of evidence needed to make appropriate judgments about the programme or policy. This evaluation will use a mixed evaluation approach which collects both qualitative and quantitative data.
Data will be collected by the evaluator at both the Oakville and Mississauga campuses of this private school. The students at the Mississauga campus are chosen because the curriculum, student body, and teaching methods are the same as the Oakville campus. The only difference is that the students at the Mississauga campus do not have access to the Reading Reinforcement program. Students from the grade 4 Mississauga class who are reading below grade level at the beginning of the year will be tracked using the same methods as the students in the Reading Reinforcement program. Program Assessments (Quantitative Data) Data based on reading ability is collected from the targeted students as well as the control sample students before, during, and after the program completion. To see if students qualify for the program, they are tested using the Developmental Reading Assessment (DRA). This is a formal assessment where the teacher chooses a selected text based on the child's instructional reading level. The teacher uses the data from the end of the previous year to determine which level to test the child. The DRA tests the child's fluency, accuracy in decoding, and comprehension skills. The DRA does not test reading stamina so this is done separately and is conducted through observations and timing how long the child can read an independent text for before they lose focus. Throughout the program, the targeted children and the control students are tested again using running records. These are less formal than the DRA but can still be done at the child's instructional level. The teacher chooses a text at the intended instructional level and has the child read appoximately 100 words. The reading reinforcement teacher (or the classroom teacher at the Mississauga campus) evaluates the fluency, decoding accuracy and asks some comprehension questions. The photo to the right is an example of a running record. The teacher continues to choose texts and tests the child until the instructional level is reached. This allows the teacher to see if the child has moved up any levels. If not, this gives the teacher some insight into areas that need work. At the same time, the classroom teacher continues to monitor the reading stamina during independent reading periods and charts the child's progress. At the end of the program, the students in both groups are re-tested again using the DRA. The classroom teachers also conduct one final reading stamina evaluation. All information is kept in Google drive and can be accessed by the classroom teacher as well as the reading reinforcement teacher. Observation of Program Implementation Data based on the number of grade 4 students who are referred to the Learning Success Centre for reading assistance is collected before the evaluation and is then collected again at the end of the school year (when referrals for the next year are made) at both the experimental and control sites. The numbers are then charted to observe any decreases in the number of grade 4 students who are referred to the Learning Resource Centre since the implementation of the Reading Reinforcement Program. If any new students enter the school in September and are referred to the Learning Resource Centre, this will be noted so that it does not interfere with the data. The evaluator will then interpret the results Quantitative Analysis The evaluator collects the DRA and running record data and summarizes it using a DRA chart. A sample of the chart is located in the sidebar to the right of this document. The evaluator uses a program such as Excel to chart the number of students referred to the LSC for reading assistance at the beginning and end of the year. The evaluator then interprets the results of the numerical data in both the experimental and control groups. The evaluator can then compare the progress of students in both groups and make conclusions and recommendations based on the data received. Anecdotal Observations (Qualitative Data) The Reading reinforcement (RR) teacher and classroom teacher are asked to keep anecdotal notes about reading behaviours noted during any reading periods at both campuses. There is no standardized set up for the anecdotal notes as each teacher completes them differently. However, the RR teacher and classroom teacher are asked to note the following:
Participant, Parent, and Teacher Surveys (Qualitative Data) Students and parents are surveyed at the beginning, middle and end of the reading reinforcement program. The targeted students will be asked questions mostly about their reading behaviours and confidence levels through a paper survey. For the students, the scale will be visual and will include happy faces, neutral faces, and sad faces to indicate the various levels. Students have filled out self-evaluation surveys similar to the one provided in the classroom setting already and so this will be familiar to them. Students at the control site will also fill out the same surveys during the same time periods. Parents will be asked similar questions about their child's reading behaviours and confidence levels at home through an email survey. The scale will be out of 4 which is the same as the rubrics levels that the parents are used to seeing come home with their children. In addition, parents will be asked questions about the reading stamina of their child. Both the classroom teacher and the RR teacher will be surveyed at the beginning, mid-point and end of the survey. Their questions will focus more around the structure of the program and the resources allocated. This survey will be a mix of open ended questions as well as scaled questions. Teachers at this school are used to filling out surveys and administration has found that teachers are more honest in writing then they are face to face so that is why this option was chosen. Qualitative Analysis The evaluator will work with the both the reading reinforcement and classroom teacher (and the classroom teacher at the control site) to read, memo, and code the qualitative data. The evaluation group will then analyze the data using the continuum of analysis. The team will move from describing the data, to interpreting it and finally will perform a critical analysis of the data. They will identify themes, sub-themes, look for patterns and comparisons of the data. Finally, they will draw conclusions from the notes. I've attached an evaluation matrix to help ensure that I've triangulated my data collection in order to answer my key evaluation questions: |
|
In an impact evaluation, it's important to look at data quality. It is important for the evaluator to ask and answer these questions:
|
Timeline
After the completion of the evaluation brief, it will be distributed to the school administrators, LSC director, Reading Reinforcement teacher, as well as any participating grade 4 teachers at the beginning of the school year. The evaluators will make it clear that the purpose of the evaluation is to conduct a summative evaluation- to see what is working in the program and to use the findings to to make decisions about future interventions. Teachers and administrators will be invited to participate in the evaluation at the onset of it. This allows the teachers and administrators to feel part of the evaluation and creates buy-in for them. They will be part of the evaluation process throughout the evaluation time period.
Together with the RR teacher and classroom teacher, the evaluator will define success of the program by creating a rubric which will define different levels of performance for each evaluative criterion. This rubric will be shared with all stakeholders. Again, this helps to create buy-in and allows stakeholders to feel included in the evaluation process.
After reviewing the plan, the evaluation will begin. Baseline data (DRA Scores) will be collected by the evaluators with assistance from the classroom teachers. Students will then be selected to attend the reading reinforcement program. During the second term (January-March), Running records, anecdotal notes, and mid-point surveys will be collected and compared to the baseline data. Finally, at the end of the year, another DRA test will be given along with final surveys. These will be collected along with more anecdotal notes and charted along with the previous data. Once all the data has been collected and analyzed, a final report and recommendations will be created.
This timeline follows the natural progression of the school year and the same reporting trajectory that the admin and teachers are used to. Since teachers are always collecting baseline data, as well as ongoing data throughout the school year, this collection will not require much additional work on the teachers part.
Together with the RR teacher and classroom teacher, the evaluator will define success of the program by creating a rubric which will define different levels of performance for each evaluative criterion. This rubric will be shared with all stakeholders. Again, this helps to create buy-in and allows stakeholders to feel included in the evaluation process.
After reviewing the plan, the evaluation will begin. Baseline data (DRA Scores) will be collected by the evaluators with assistance from the classroom teachers. Students will then be selected to attend the reading reinforcement program. During the second term (January-March), Running records, anecdotal notes, and mid-point surveys will be collected and compared to the baseline data. Finally, at the end of the year, another DRA test will be given along with final surveys. These will be collected along with more anecdotal notes and charted along with the previous data. Once all the data has been collected and analyzed, a final report and recommendations will be created.
This timeline follows the natural progression of the school year and the same reporting trajectory that the admin and teachers are used to. Since teachers are always collecting baseline data, as well as ongoing data throughout the school year, this collection will not require much additional work on the teachers part.
Reporting Strategies
A collaborate approach has been implemented since the beginning of the evaluation. The collaborative approach will ensure that the report is tailored to meet the needs of the stakeholders as the evaluator and he teachers have worked together throughout the evaluation process.
The evaluators will communicate their findings with the primary stakeholders of the Reading Reinforcement Program (Reading Reinforcement teacher, administration and classroom teacher). The evaluators will communicate findings in person, through email, and through a written report with the primary stakeholders.
The report will be clear, concise and will focus on the key evaluation questions. The report will state evaluative answers to the key evaluation questions. Examples will be provided and charts will show how the data the was interpreted. The language will be clear and concise and the explanations should be understood by anyone reading the report. The rubrics will be included as well. (Better Evaluation Website)
The stakeholders will be invited to a meeting during school hours. The report will be shared via email to stakeholders prior to the meeting. This will allow them to read the report and come prepared with questions and comments. This will keep the meeting focused and concise. Teachers will be given release time from classroom duties in order to attend. Having the meeting during school hours allows teachers to maintain their work/life balance and they will be more engaged in the reporting process. During this meeting, the findings will be shared with stakeholders and recommendations will be made. The primary stakeholders will get a chance to ask questions and then work together with the evaluator to create a plan for recommendation implementation. This form of communication follows the Utility Standards or Evaluation to serve and inform the needs of its intended users as it is appropriate and beneficial to both the evaluators and the stakeholders.
After the report meeting, the students will be given access to the parts of the report that affect them in a way that they understand. The Reading Reinforcement teacher will share the strengths of the program with them and also share the areas of improvement for the next year.
Parents of the students in the program will be given a concise report with the strengths and action plan for the future which will be emailed to them. This method of communication was chosen as it is familiar to parents and fits with the model of communication that the school follows.
The evaluators will communicate their findings with the primary stakeholders of the Reading Reinforcement Program (Reading Reinforcement teacher, administration and classroom teacher). The evaluators will communicate findings in person, through email, and through a written report with the primary stakeholders.
The report will be clear, concise and will focus on the key evaluation questions. The report will state evaluative answers to the key evaluation questions. Examples will be provided and charts will show how the data the was interpreted. The language will be clear and concise and the explanations should be understood by anyone reading the report. The rubrics will be included as well. (Better Evaluation Website)
The stakeholders will be invited to a meeting during school hours. The report will be shared via email to stakeholders prior to the meeting. This will allow them to read the report and come prepared with questions and comments. This will keep the meeting focused and concise. Teachers will be given release time from classroom duties in order to attend. Having the meeting during school hours allows teachers to maintain their work/life balance and they will be more engaged in the reporting process. During this meeting, the findings will be shared with stakeholders and recommendations will be made. The primary stakeholders will get a chance to ask questions and then work together with the evaluator to create a plan for recommendation implementation. This form of communication follows the Utility Standards or Evaluation to serve and inform the needs of its intended users as it is appropriate and beneficial to both the evaluators and the stakeholders.
After the report meeting, the students will be given access to the parts of the report that affect them in a way that they understand. The Reading Reinforcement teacher will share the strengths of the program with them and also share the areas of improvement for the next year.
Parents of the students in the program will be given a concise report with the strengths and action plan for the future which will be emailed to them. This method of communication was chosen as it is familiar to parents and fits with the model of communication that the school follows.
Program Evaluation Standards
Program Impact
This program evaluation is intended to positively impact all stakeholders in the Reading Reinforcement program. The wide range of qualitative and quantitative data collected and analyzed (as shown in the evaluation matrix) will provide evaluators with the information necessary to examine the program in-depth.
The stakeholders are involved throughout the process as they are the experts on the program. Including them in the evaluation ensures that the findings are accurate, valid and beneficial for the program users. Including them also allows them to learn about program evaluations so that they can continue to evaluate programs on their own. The evidence based findings will help evaluators answer the key evaluation questions and provide valuable feedback in terms of the strengths and areas of improvement for the Reading Reinforcement program. The action plan will include a list of recommendations that will be able to be implemented for the following year in order to enhance the program. The improvements made will directly impact the success of the students involved and of the future students who may also be struggling with reading. |
References
1. Better Evaluation Website- Impact Evaluation: https://www.betterevaluation.org/en/themes/impact_evaluation
2. Data Collection and Analysis for Impact evaluation Video: https://www.youtube.com/watch?v=HFGVJJMDo4I
3. Standards for Program Evaluation: http://www.oecd.org/dev/pgd/38406354.pdf
4. Qualitative Data Analysis StAR Toolkit, ONQ Qualitative Data Analysis StAR_Toolkit.pdf
2. Data Collection and Analysis for Impact evaluation Video: https://www.youtube.com/watch?v=HFGVJJMDo4I
3. Standards for Program Evaluation: http://www.oecd.org/dev/pgd/38406354.pdf
4. Qualitative Data Analysis StAR Toolkit, ONQ Qualitative Data Analysis StAR_Toolkit.pdf