Evaluation criteria

Cases studies Explored databases Electronic search strategies Screening criteria Evaluation criteria Structure of summaries Idiomatic translation

Different types of checklists are used for the different types of studies. Checklists are used as reminders of the issues that each reviewer must evaluate. A score point system is not used. Every item on the ckecklists is classified as "yes", "no", "don't know" or "not applicable".

The checklists used for each type of study are:

For articles on studies of frequency, case-series, risk factors, and prognostic factors:

(Developed by the Editorial Committee of the Professional Site of the Web of the Back taking into account the recommendations of the University of York)
(http://www.york.ac.uk/inst/crd)

1. Does the study incur in any of the following biases in a way that validity of results may be questioned?
   1. Reading Bias: For example, bibliographic references bias: Authors only include references supportive of their point of view.
   2. Sample Definition or Selection Bias: For example, bias of diagnostic accessibility, hospitalization rate, Berkson's bias or non-respondent bias.
   3. Exposure Bias: For example, adherence to treatment bias.
   4. Measuring Bias: For example, bias involving memory, instruments, or non-sensitive measurements.
   5. Analysis Bias: For example, exhaustive search for correlations without a previous hypothesis.
   6. Interpretation Bias: For example, confusing statistical and biological significance or equating correlation to causality.
2. Are the subjects appropriate for the study objective? (If there are several groups, it should be applied to all and each one of them)
3. Are the variables appropriate for the study objective?
4. Have the variables been measured adequately?
5. If applicable, is the follow-up period adequate?
6. If the study includes follow-up of subjects, and there is loss to follow-up, are the results still valid despite this?
7. Has the study objective been clearly described?
8. Are the studied population's characteristics adequate for the established objective? (Temporal and geographic characteristics, sampling technique, etc.)
9. If more than one group is studied, are the inclusion criteria adequate for each group?
10. Is the sample size adequate for the analysis performed?
11. Are the described subjects' characteristics relevant and given in enough detail?
12. Are the described sample population's characteristics relevant and sufficient?
13. If subjective variables were used (for example, interpretation of a radiology test or clinical records), is there a reference to its inter-observer variability or is it analyzed?
14. If a measurement system or a rating scale was used on the relevant variables, is there a comment or reference to its accuracy? (If all are considered to be standard, the answer is "yes").
15. If applicable, have the interpretations of the different evaluations and measurements been performed under blind conditions?
16. Was the performed statistical analysis appropriate for the established objective? (For example, in association studies, have the potential confounding variables been accounted for?).
17. Are the results presented in adequate detail?
18. Are conclusions consistent with the presented results?

Articles concerning treatments:

(Modified by the Editorial Committee of the Professional Site of the Web of the Back, based on Koes BW, Scholten RJ, Mens JM, Bouter LM. Efficacy of non-steroidal anti-inflammatory drugs for low back pain: a systematic review of randomized clinical trials. 1997; 56(4):214-23)

1. Are the patients' selection criteria specified?
2. Treatment Allocation:
   1. Has treatment been assigned under random conditions?
   2. Has treatment allocation been performed under blind conditions?
3. Were treatment groups sufficiently similar concerning the most relevant prognostic factors? If the potential difference was taken into consideration, answer "yes".
4. Are the study and control interventions explicitly described?
5. Was the treatment provider blind to the intervention?
6. Are co-interventions sufficiently comparable? If they were avoided, answer yes.
7. Is there a sufficient grade of baseline homogeneity between all groups?
8. Were patients blinded to the assigned intervention?
9. Was outcome assessment blinded to interventions?
10. Are the measured variables the relevant ones for the study?
11. Are adverse events described?
12. Is the study dropout rate described and acceptable?
13. Chronography of the follow-up measurements:
    1. Was a short-term follow-up measurement performed?
    2. Was a long-term follow-up measurement performed
14. Was the time of evaluation of results comparable for all groups?
15. Is the sample size of each group described?
16. Was an intention-to-treat analysis performed?
17. Are estimators of dispersion or variability for the main outcomes presented?

For articles of clinical-diagnostic correlation:

(Prepared by the Editorial Committee of the Professional Site of the Web of the Back taking into account the recommendations of the University of York)
(http://www.york.ac.uk/inst/crd)

1. Are there at least two established groups to allow comparison (symptomatic/asymptomatic, with/without findings)?
2. Have the tests and/or assessments been performed appropriately?
3. Is there an appropriate reference standard?
4. Was the follow-up period appropriate?
5. In case of withdrawals, are results still valid despite them?
6. Is the proposed diagnostic test to be used properly described?
7. Are the studied population's characteristics appropriate?
8. Were inclusion and exclusion criteria adequate? (If they have not been described, answer "No").
9. Was the sample size of the groups adequate for the analysis performed?
10. Are patients' characteristics described?
11. Is the population source described?
12. Were the test results and evaluations clearly classified?
13. If a specific measurement/classification was used, was the test precision described or referenced? (If all are standards, answer "yes").
14. Were interpretations of reference standards and tests blinded?
15. Was the statistical analysis performed appropriate for the established objective?
16. Are results presented in adequate detail?
17. Were results interpreted correctly?

For Diagnostic Studies:

(Modified by the Editorial Committee of the Professional Site of the Web of the Back, based on Mulrow et al. Assesing quality of a diagnostic test evaluation. Journal of General Internal Medicine. 1998;4:288-95)

1. Were diseased and nondiseased patients included?
2. Was the test appropriately performed?
3. Was there an appropriate reference standard?
4. Was the proposed use described?
5. Was the appropriate population studied?
6. Were inclusion and exclusion criteria described?
7. Was a wide spectrum of diseased patients included?
8. Were control (nondiseased) patients with comorbid diseases included?
9. Was the sample size adequate?
10. Were patients' characteristics described?
11. Were case (diseased) patients with comorbid diseases included?
12. Was the population source described?
13. For the diagnostic test, were the terms normal/abnormal clearly defined?
14. Was test precision described?
15. Were interpretations of reference standards and tests blinded?
16. Was the reference standard appropriately performed?
17. As far as the reference standard is concerned, were the terms normal/abnormal defined?
18. Are data presented in adequate detail?
19. Were uninterpretable results enumerated?

For systematic literature reviews and meta-analyses:

(Modified by the Editorial Committee of the Professional Site of the Web of the Back, based on Oxman AD, Guyatt GH. Guideliness for reading literature reviews CMAJ. 1988;138:698-703)

1. Is the objective of the review precisely stated?
2. Does the source selection guarantee an exhaustive review?
3. Were electronic search strategies used appropriately to locate relevant articles?
4. Among the located articles, were explicit methods used to determine which articles to include and which not in the review?
5. Are these explicit methods appropriate for the objective of the review? (Answer: "Not known", if methods are not described).
6. Was the publication bias estimated?
7. Was methodological quality of the included studies assessed?
8. Was methodological assessment correct? (For example using a previously validated, or at least a complete, quality rating scale).
9. Were authors consulted for doubts about their articles?
10. Is the study replicable? (Concerning the method used, independently of whether it would obtain the same results or not).
11. Have measures been taken to control bias during the review process of the included articles? (For example, various independent reviewers for each article).
12. Are results of each of the included studies appropriately presented?
13. Was variation in the findings of the relevant studies analyzed?
14. If there is no "evidence of the effect", is "evidence of no effect" differentiated?
15. In meta-analyses, was heterogeneity of the relevant studies evaluated? (Concerning methodology, study population, variables, measuring devices, etc).
16. If the studies included were heterogeneous, are results presented in subgroups?
17. In meta-analyses, were the findings of the primary studies combined appropriately?
18. If applicable, were sensitivity studies performed?
19. Are guidelines for further reviews recommended?
20. Were the reviewers' conclusions supported by the data cited?

For studies on economic aspects:

(Prepared by the Editorial Committee of the Professional Site of the Web of the Back based on M. F. Drummond "Principles of economic appraisal in health care" and McMaster Health Science Center "How to read clinical journals: VII: To understand an economic evaluation (Parts A and B) Can. Med. Asso. J. 1984).

1. Is the study objective clearly stated?
2. Are objectives quantifiable?
3. Is it clearly specified if the study deals at a micro or a macro level?
4. Does it clearly mention from whose (hospital, insurance company, society, government, etc.) perspective the study was carried out?
5. Was an important alternative to the proposed technology omitted? (For example, results without applying the technology, placebo, other technology, changes in lifestyle, no intervention).
6. Were costs and consequences of the proposed technology analyzed?
7. Were several and relevant alternatives compared?
8. Was economic assessment thoroughly performed?
9. Were relevant levels at which costs and outcomes are produced clearly identified? If costs and outcomes were estimated, were the assumptions clearly presented?
10. Was the calculation method explicitly stated?
11. Were measurement units clearly defined?
12. Were appropriate quantitative (or qualitative) values assigned to the different analysis points?
13. Were mean and marginal costs and consequences differentiated?
14. Was a consideration made of the time factor on costs and benefits?
15. Was the use of the discount rate applied to this consideration justifiable?
16. Were the value ranges of the relevant variables in the sensitivity analysis justified?
17. Was sensitivity of results to changes in the variables considered?
18. Are the conclusions consistent with the results?
19. Are the results discussed in a useful manner for decision-makers?
20. Are the results discussed exhaustively?

Quality Control Mechanisms:

• 20% of all articles is automatically submitted to at least two reviewers, and the consistency of their evaluations is assessed.
• Along with this systematic method, articles are sent to at least two reviewers when:
  1. The Editorial Committee foresees there may be a risk of inconsistent evaluations.
  2. Any reviewer considers that a review should be performed by more than one evaluator.
• All reviewers meet periodically to guarantee the best possible homogeneity during the review process.
• The assessment and methodology criteria are subjected to the international scientific community through publication in scientific journals and discussion in forums involved in systems of scientific evidence assessment. Review criteria are revised periodically according to the reviewers', inputs and the updating of the criteria of the international scientific community.
• The Editorial Committee systematically reviews 20% of the articles assessed, regardless of whether they were approved or discarded.