Integrity and Reliability

Professional Tools and Resources for Behavior Analysts

hosted at Science of Behavior – www.scienceofbehavior.com

This page is a work in progress.
Two forms of reliability are important for assessing the effectiveness of an intervention:
    • Intervention reliability
    • Data reliability

Intervention Reliability

Federal laws and Florida DOE rules require educators to assess a student’s Response to Intervention (RTI) through repeated measurement of two key outcomes:

1) Intervention reliability – the extent to which an intervention is fully and correctly implemented on repeated trials

2) Student progress – measures of the student’s behavior(s) or skills(s)

  • Traditionally, educators have primarily relied on the second measure, student progress.
  • Inadequate student progress was described when goals were not being met and/or there was a significant gap between the progress of the student and typical peers.
  • In the past, inadequate student progress alone may have been the basis for a formal evaluation and eligibility determination for special education services.
  • However, to properly assess RTI, we now must show evidence of the student’s inadequate progress, plus evidence that the intervention was fully and properly implemented (i.e., intervention reliability).

Intervention reliability (also known by terms such as intervention fidelity, procedural reliability or treatment integrity) is a relatively new issue in many schools. However, a large body of scientific research and related professional literature has demonstrated that high levels of intervention reliability may be vital for assuring the progress of persons with behavioral or learning challenges.

  • Intervention reliability should be assessed across all tiers of instructional and behavioral interventions. “Walk throughs” are commonly used in some schools for checking intervention reliability for Schoolwide and Classroom/Targeted Group interventions.
  • Two methods are generally necessary to achieve high levels of intervention reliability:

1) effective training (with related documentation), and

2) ongoing monitoring to assure consistent, correct procedure implementation throughout the intervention (with related documentation).

  • The following are examples of each method:

Effective training – an example:

1 – A) Prior to the training, use a checklist format to describe the essential elements or steps of procedures in the intervention.

  • When standard procedures are used the steps may be taken from established checklists.
    • Sample: Precision Commands checklist
  • When individualized procedures are used (e.g., in a behavioral intervention plan), the steps may be taken from the plan.
  • Major advantages of using the checklist are its bonus benefits as a tool for self-checking interventions and observations by others when monitoring intervention integrity.

1 – B) Determine the minimum criteria for proficient performance by the person being trained.

  • Interventions that involve a degree of risk may have higher criteria.
  • Example of higher criteria:
    • 100% correct, independent natural/live performance for 3 consecutive attempts, (in the classroom with the student, no checklist or trainer tips provided).
  • Example of lesser criteria:
    • 80% correct role play performance for 2 consecutive attempts, (referring to a written checklist or receiving trainer tips is acceptable).

2) Begin the training with a video, live, or simulated (role play) demonstration of the intervention.

  • Natural, live demonstrations are highly preferred, when practical.
  • The intervention may be best demonstrated by the person(s) who wrote it.
  • Sample video of a skill demonstration: Precision Commands

3) Discuss and explain the key steps. As needed, demonstrate those steps again and emphasize important features.

  • Frequently refer to the related steps in the checklist.

4) The person being trained demonstrates the procedure in a natural, live situation (when practical).

  • In some cases, it is better to role play the procedure until proficiency is demonstrated (see below), before attempting a live demonstration.
  • It may also be helpful to allow the person being trained to refer to the checklist during this demonstration.

5) As the steps are being performed, the trainer gives specific feedback and records notes on the checklist.

6) When the demonstration is completed, the trainer reviews the checklist and gives praise to shape up peak performance.

  • The trainer calculates the percentage of correct responses and gives feedback.
  • In the first demonstrations, the person being trained may refer to the checklist.
  • Finally, the person being trained may demonstrate the skill independently (without the checklist or tips from the trainer).

7) When the criteria for proficient performance is not met, extra training and demonstration opportunities are provided until the person succeeds in meeting or exceeding the criteria.

  • Remember: The point of the training is to shape up performance by reinforcing gradual improvements.
  • If a pattern of unsuccessful demonstrations occurs, the entire training plan and checklist
  • should be reviewed by appropriate team members. When needed, consult with professional trainers.

8) In many cases, it is important to record the training results.

  • When such documentation may cause sensitivity by the person being trained, consult with the site administrator. It may be beneficial to engage the administrator in the training and recording procedures.
  • It also may be preferred if only the performance outcomes at or above the criteria are recorded.
  • Performance outcomes below the proficiency criteria may be recorded as “incomplete” (pending additional training).

Ongoing monitoring – an example:

  • Training (as described above) helps others to learn or acquire the skills needed to correctly implement the intervention.
  • Monitoring helps others become fluent or automatic in the correct implementation of the intervention each time it is repeated.
  • Monitoring also helps prevent procedure “drift” that naturally occurs if there is no periodic review of the procedure.
  • Monitoring should provide many opportunities to reinforce correct performance!
  • When needed, reminders are given to help the person maintain peak performance.
  • Plan for multiple persons to monitor the intervention. Each person must be proficient in the intervention. These persons may include: Administrators; School Psychologists; Social Workers; Guidance Counselors; ESE Specialists; Behavior Analysts, Coaches, or Specialists; Reading or Intervention Specialists; Mentors; Team Leaders; and others.
  • Determine an initial schedule and criteria for fading the frequency of monitoring.
  • Complex interventions, and/or those that involve some degree of risk, may require more frequent monitoring.
  • Example of a schedule for fading the frequency of monitoring:
    • Begin with monitoring 3 times weekly.
    • When the criteria for proficient performance is met 3 consecutive times, fade monitoring to 2 times weekly.
    • When the criteria is met 2 more consecutive times, fade monitoring to once weekly.
    • When the criteria is met 2 more consecutive times, fade monitoring to twice monthly.
    • When the criteria is met 2 more consecutive times, fade monitoring to once monthly.

1) Use the same checklist format that was used during training (see above).

2) Plan the best time to observe the person demonstrate the procedure in a natural, live situation.

  • Natural observations are highly preferred. However, when there are rare opportunities for the person to naturally use the procedures, observe a role play (rehearsal).
  • For procedures that are used continually or frequently, randomly scheduled “walk throughs” may be planned if they are not too disruptive.
  • If appropriate, provide an extra copy of the checklist to the person during the monitoring.

3) As the steps are being performed, the observer gives specific feedback and records notes on the checklist. If frequent feedback disrupts the situation, the observer withholds feedback until the end of the demonstration.

4) When the demonstration is completed, the observer reviews the checklist and gives reminders, if needed, and praises correct or improved performance.

  • The observer calculates the percentage of correct responses and gives feedback to the educator.
  • Use the same criteria for proficient performance used during the training (see above).

5) When the criteria for proficient performance is not met, provide immediate (re)training and demonstration opportunities, or plan time for (re)training and demonstration opportunities ASAP.

  • If a pattern of unsuccessful demonstrations occurs, the entire training plan, monitoring plan, and checklist should be reviewed by appropriate team members. When needed, consult with professional trainers.

6) In many cases, it is important to record the monitoring results.

  • When such documentation may cause sensitivity, consult with the site administrator.
  • It may be beneficial to engage the administrator in the monitoring and recording procedures.
  • It also may be preferred if only the performance outcomes at or above the criteria are recorded.
  • Performance outcomes below the proficiency criteria may be recorded as “incomplete” (pending additional training).

7) Transfer the percentage of correct responses to a graph. Review the graph when giving feedback and setting goals with the educator.

Data reliability

Data reliability is sometimes described as the extent to which a measurement procedure yields the same result on repeated trials.      

Inter-observer reliability (IOR) is a specific concern for data recorded in school settings. IOR is usually assessed by comparing the judgments of two observers who independently and simultaneously used the same procedure to measure a well-defined target behavior. The purpose is to see how closely they agreed or disagreed on the occurrence of the target behavior (Cooper, Heron & Heward, 1987). This important topic is addressed in more detail under the “Measure” tab.???