Rating of Randomised Controlled Trials in OTseeker
Rigorous answers to questions about treatment effectiveness can be provided by properly designed randomised controlled trials. Unfortunately the literature contains both well performed trials which draw valid conclusions and trials that have sufficient bias so as to impact on their conclusions. The reader must be able to distinguish between the two. A number of guides or scales are available to help the reader consider important features (or criteria) of randomised controlled trials, that if implemented well, can reduce the risk of bias. One of these is the PEDro scale containing 10 such criteria, developed by members of the team who developed PEDro - the Physiotherapy Evidence Database.
OTseeker has separated the PEDro scale into two concepts: 1) internal validity (8 items) and 2) statistical reporting (2 items), due to the different natures of these concepts. The scale used in OTseeker is therefore referred to as the PEDro scale (partitioned). The PEDro Scale (Partitioned) can be downloaded for use in teaching, research or clinical practice. There is also a PEDro Scale (Partitioned) Rating Guideline that explains each criterion in detail.
To assess key features of internal validity we look for unambiguous confirmation of 8 criteria, including random allocation, concealment of allocation, comparability of groups at baseline, blinding of patients, therapists and assessors, analysis by intention to treat and adequacy of follow-up. For each trial, OTseeker indicates which of these criteria have been met. The best interpretation of the information provided is to consider the potential impact the presence or absence of these features might have on the trial, rather than relying on the number of items met.
To assess statistical reporting we look for 2 criteria including between-group statistical comparisons and reports of both point estimates and measures of variability.
These criteria are rated on the basis of what they report - if a trial does not report that a particular criterion was met, it is scored as if the criterion was not met ('guilty till proven innocent').
The PEDro scale (partitioned) has been used to rate approximately 7,500 trials in OTseeker (up until end of June 2013). Given the rapid increase in the number of randomised controlled trials being published relevant to occupational therapy, it will not be possible for the OTseeker team to rate all new trials that are entered into the database after June 2013. However, we will seek to provide enhanced resources for therapists to be able to rate trials themselves and provide ratings for some trials where possible. The details about the rating for the 7,500 randomised controlled trials that already been rated in OTseeker will continue to be available. These details can be found underneath the citation for each randomised controlled trial on the Search Results pages of OTseeker, and in more detail on the Detailed Results Pages.
The criteria for each trial are rated by two raters independently of each other. If the ratings of the two raters differ, a third rater rates the trial and resolves any disagreements. On the Detailed Search Results page, the rating status of the trial will display 'This rating has not yet been confirmed' if 1) only one rater has rated the trial so far or 2) there was disagreement between the two raters and a third rater is yet to rate the trial. Once rating consensus has been achieved, the rating status of the trial will display 'This rating has been confirmed'. Some trials on OTseeker have not been rated and this is indicated by ‘This trial has not been rated’.
Ratings are undertaken by members of the OTseeker team and/or research research assistants. All raters have undergone training in the use of PEDro scale (partitioned). A study that examines the reliability of the PEDro scale (partitioned) found good inter-rater reliability.
Tooth, L., Bennett, S., McCluskey, A., Hoffmann, T., McKenna, K., & Lovarini, M. (2005). Appraising the quality of randomized controlled trials: Inter-rater reliability for the OTseeker evidence database. Journal of Evaluation in Clinical Practice, 11, 547-555.