Choose a Quality Review Tool by Study Design for an Evidence Table
A quality review tool is the checklist you use to judge how much a study result can be trusted. Choose it from the study design, then keep each domain judgment in the evidence table. Randomized trials use RoB 2. Non-randomized studies of interventions use ROBINS-I. Cohort and case-control studies often use the Newcastle-Ottawa Scale. Diagnostic accuracy studies start from QUADAS-2; the developers now recommend QUADAS-3 for a new review. Systematic reviews of interventions use AMSTAR 2. CASP publishes checklists for several other designs. The main guide is Quality assessment on SystematicReviewTools.app.
Figure. Match the design, then record the tool. This figure is not a link. The sources are in the sections below.
What to put in the evidence table
Use one row per result you will synthesize, not one row that pretends a whole paper has a single bias judgment. RoB 2 and ROBINS-I are result-level tools: the same paper can differ by outcome. Columns that stay usable in a synthesis are:
- study label and design, in the paper’s words
- the outcome, timepoint, and effect estimate this row is about
- the tool name and version
- one column per domain judgment
- a short support-for-judgment note, preferably a quote and a location
That is the same job as choosing the tool. The quality assessment guide is the longer walkthrough of how to embed the framework in a review. This page is the map from design to tool, with the citations next to each name.
Randomized trials: RoB 2
Cochrane Handbook chapter 8 describes RoB 2 for individually randomized parallel-group trials. The five domains are required, and the chapter says no extra domains should be added:
- bias arising from the randomization process
- bias due to deviations from intended interventions
- bias due to missing outcome data
- bias in measurement of the outcome
- bias in selection of the reported result
Each domain is judged Low, Some concerns, or High. The overall judgment for the result follows the least favourable domain, with the chapter’s rule that some concerns in several domains can become High overall. The tool paper is Sterne and colleagues, BMJ 2019. The RoB 2 site also publishes versions for cluster-randomized trials and crossover trials.
In the evidence table, store the five domain judgments and the overall judgment on the row for that outcome. Do not average them into a score. Chapter 8 says the signalling questions lead to a proposed judgment, and reviewers can override it if they justify the change.
Non-randomized studies of interventions: ROBINS-I
ROBINS-I (Sterne and colleagues, BMJ 2016) is the domain-based tool for non-randomized studies of interventions. The paper describes seven domains: confounding and selection of participants into the study before the intervention starts; classification of the interventions; then, after the intervention starts, deviations from intended interventions, missing data, measurement of outcomes, and selection of the reported result.
Judgments are Low, Moderate, Serious, or Critical. The paper says Low corresponds to the risk of bias in a high-quality randomized trial, and that an observational study will only rarely be Low on confounding. Use ROBINS-I when the review question is the effect of an intervention and the study was not randomized. A cohort or case-control study that is not asking that causal question is a different choice, covered next. The profile switched on in Evidence Table Builder is the ROBINS-I V2 follow-up cribsheet, described with the other live profiles below.
Cohort and case-control studies: Newcastle-Ottawa
The Newcastle-Ottawa Scale (Wells and colleagues, Ottawa Hospital Research Institute) assesses non-randomized studies for use in a meta-analysis with a star system. The page says a study is judged on three perspectives: selection of the study groups; comparability of the groups; and ascertainment of the exposure (case-control) or the outcome (cohort).
Stars are not the same thing as a RoB 2 domain judgment. If you use the scale, keep the three perspectives in separate columns and say in the protocol how a star total will, or will not, change the synthesis. ROBINS-I is the better fit when the row is an intervention effect and you need domain-level bias judgments rather than stars.
Diagnostic accuracy: QUADAS-2 and QUADAS-3
QUADAS-2 is the previous version of the diagnostic-accuracy tool. The University of Bristol QUADAS project says QUADAS-3 is the version it recommends for systematic reviews of test accuracy for diagnosis, screening, or staging. The project developed QUADAS-3 from feedback on QUADAS-2, and it assesses risk of bias and applicability at the estimate level rather than once for the whole study. The citation the project prints is Whiting and colleagues, Annals of Internal Medicine, 17 February 2026, doi:10.7326/ANNALS-25-02104.
The explanation and elaboration describes four domains: participants, index test, target condition, and analysis. Risk of bias in a domain is low, high, or insufficient information. An evidence table of diagnostic accuracy should therefore have one row per accuracy estimate (for example a sensitivity and specificity pair), the tool version (QUADAS-2 or QUADAS-3), and a column per domain. Say which version the protocol locked. A review that already extracted QUADAS-2 should not be silently relabeled as QUADAS-3.
Systematic reviews of interventions: AMSTAR 2
AMSTAR 2 (Shea and colleagues, BMJ 2017) appraises systematic reviews that include randomized trials, non-randomized studies of interventions, or both. The paper says the instrument has 16 items, is not intended to generate an overall score, and reaches an overall confidence rating from weaknesses in critical domains.
Use it when the included “study” is itself a systematic review of interventions. ROBIS is a different review-level tool aimed at risk of bias in a systematic review; AMSTAR 2 is the methodological-quality instrument described in that BMJ paper. Pick one as the primary tool for that row and name it in the table.
Qualitative studies and other designs: CASP
The Critical Appraisal Skills Programme publishes free checklists by study type. The page lists checklists for systematic reviews, randomized trials, qualitative studies, cohort studies, diagnostic studies, case-control studies, economic evaluations, clinical prediction rules, and cross-sectional studies, plus separate checklists for systematic reviews with meta-analysis of trials and of observational studies.
CASP is a set of design-specific checklists, not one score you can drop into a meta-analysis. For a qualitative evidence table, use the qualitative checklist and keep each answer (yes, no, or can’t tell, as that checklist uses) in its own column. Do not force RoB 2’s Low / Some concerns / High labels onto a qualitative study.
What Evidence Table Builder runs today
Evidence Table Builder’s quality-assessment workspace is separate from the extraction table. You choose a profile, upload the PDFs, and download the run as a spreadsheet or JSON. The profiles switched on in the current release are:
- RoB 2 for parallel, cluster, and crossover randomized trials
- ROBINS-I V2 for a follow-up study of a non-randomized intervention
- Newcastle-Ottawa for cohort and case-control studies
- AMSTAR 2 and ROBIS for systematic reviews
- six NIH National Heart, Lung, and Blood Institute checklists (controlled intervention, observational cohort and cross-sectional, case-control, before-after, case series, and systematic reviews), from the NHLBI study-quality tools
- CORE extraction, which collects study facts and does not produce a risk-of-bias judgment
QUADAS-2, QUADAS-3, and the CASP checklists are not switched on there. Record those in the evidence table using the columns above. The workspace prepares a draft with quoted passages from the PDF for the profiles that are on; the judgment you export is still yours to confirm. A longer tour of those profiles is in Quality assessment in Evidence Table Builder.
Related reading
A quality review tool is the checklist you use to judge how much a study result can be trusted. Choose it from the study design, then keep each domain judgment in the evidence table. Randomized trials use RoB 2. Non-randomized studies of interventions use ROBINS-I. Cohort and case-control studies often use the Newcastle-Ottawa Scale. Diagnostic accuracy studies start from QUADAS-2; the developers now recommend QUADAS-3 for a new review. Systematic reviews of interventions use AMSTAR 2. CASP publishes checklists for several other designs. The main guide is Quality assessment on SystematicReviewTools.app.
Figure. Match the design, then record the tool. This figure is not a link. The sources are in the sections below.
What to put in the evidence table
Use one row per result you will synthesize, not one row that pretends a whole paper has a single bias judgment. RoB 2 and ROBINS-I are result-level tools: the same paper can differ by outcome. Columns that stay usable in a synthesis are:
- study label and design, in the paper’s words
- the outcome, timepoint, and effect estimate this row is about
- the tool name and version
- one column per domain judgment
- a short support-for-judgment note, preferably a quote and a location
That is the same job as choosing the tool. The quality assessment guide is the longer walkthrough of how to embed the framework in a review. This page is the map from design to tool, with the citations next to each name.
Randomized trials: RoB 2
Cochrane Handbook chapter 8 describes RoB 2 for individually randomized parallel-group trials. The five domains are required, and the chapter says no extra domains should be added:
- bias arising from the randomization process
- bias due to deviations from intended interventions
- bias due to missing outcome data
- bias in measurement of the outcome
- bias in selection of the reported result
Each domain is judged Low, Some concerns, or High. The overall judgment for the result follows the least favourable domain, with the chapter’s rule that some concerns in several domains can become High overall. The tool paper is Sterne and colleagues, BMJ 2019. The RoB 2 site also publishes versions for cluster-randomized trials and crossover trials.
In the evidence table, store the five domain judgments and the overall judgment on the row for that outcome. Do not average them into a score. Chapter 8 says the signalling questions lead to a proposed judgment, and reviewers can override it if they justify the change.
Non-randomized studies of interventions: ROBINS-I
ROBINS-I (Sterne and colleagues, BMJ 2016) is the domain-based tool for non-randomized studies of interventions. The paper describes seven domains: confounding and selection of participants into the study before the intervention starts; classification of the interventions; then, after the intervention starts, deviations from intended interventions, missing data, measurement of outcomes, and selection of the reported result.
Judgments are Low, Moderate, Serious, or Critical. The paper says Low corresponds to the risk of bias in a high-quality randomized trial, and that an observational study will only rarely be Low on confounding. Use ROBINS-I when the review question is the effect of an intervention and the study was not randomized. A cohort or case-control study that is not asking that causal question is a different choice, covered next. The profile switched on in Evidence Table Builder is the ROBINS-I V2 follow-up cribsheet, described with the other live profiles below.
Cohort and case-control studies: Newcastle-Ottawa
The Newcastle-Ottawa Scale (Wells and colleagues, Ottawa Hospital Research Institute) assesses non-randomized studies for use in a meta-analysis with a star system. The page says a study is judged on three perspectives: selection of the study groups; comparability of the groups; and ascertainment of the exposure (case-control) or the outcome (cohort).
Stars are not the same thing as a RoB 2 domain judgment. If you use the scale, keep the three perspectives in separate columns and say in the protocol how a star total will, or will not, change the synthesis. ROBINS-I is the better fit when the row is an intervention effect and you need domain-level bias judgments rather than stars.
Diagnostic accuracy: QUADAS-2 and QUADAS-3
QUADAS-2 is the previous version of the diagnostic-accuracy tool. The University of Bristol QUADAS project says QUADAS-3 is the version it recommends for systematic reviews of test accuracy for diagnosis, screening, or staging. The project developed QUADAS-3 from feedback on QUADAS-2, and it assesses risk of bias and applicability at the estimate level rather than once for the whole study. The citation the project prints is Whiting and colleagues, Annals of Internal Medicine, 17 February 2026, doi:10.7326/ANNALS-25-02104.
The explanation and elaboration describes four domains: participants, index test, target condition, and analysis. Risk of bias in a domain is low, high, or insufficient information. An evidence table of diagnostic accuracy should therefore have one row per accuracy estimate (for example a sensitivity and specificity pair), the tool version (QUADAS-2 or QUADAS-3), and a column per domain. Say which version the protocol locked. A review that already extracted QUADAS-2 should not be silently relabeled as QUADAS-3.
Systematic reviews of interventions: AMSTAR 2
AMSTAR 2 (Shea and colleagues, BMJ 2017) appraises systematic reviews that include randomized trials, non-randomized studies of interventions, or both. The paper says the instrument has 16 items, is not intended to generate an overall score, and reaches an overall confidence rating from weaknesses in critical domains.
Use it when the included “study” is itself a systematic review of interventions. ROBIS is a different review-level tool aimed at risk of bias in a systematic review; AMSTAR 2 is the methodological-quality instrument described in that BMJ paper. Pick one as the primary tool for that row and name it in the table.
Qualitative studies and other designs: CASP
The Critical Appraisal Skills Programme publishes free checklists by study type. The page lists checklists for systematic reviews, randomized trials, qualitative studies, cohort studies, diagnostic studies, case-control studies, economic evaluations, clinical prediction rules, and cross-sectional studies, plus separate checklists for systematic reviews with meta-analysis of trials and of observational studies.
CASP is a set of design-specific checklists, not one score you can drop into a meta-analysis. For a qualitative evidence table, use the qualitative checklist and keep each answer (yes, no, or can’t tell, as that checklist uses) in its own column. Do not force RoB 2’s Low / Some concerns / High labels onto a qualitative study.
What Evidence Table Builder runs today
Evidence Table Builder’s quality-assessment workspace is separate from the extraction table. You choose a profile, upload the PDFs, and download the run as a spreadsheet or JSON. The profiles switched on in the current release are:
- RoB 2 for parallel, cluster, and crossover randomized trials
- ROBINS-I V2 for a follow-up study of a non-randomized intervention
- Newcastle-Ottawa for cohort and case-control studies
- AMSTAR 2 and ROBIS for systematic reviews
- six NIH National Heart, Lung, and Blood Institute checklists (controlled intervention, observational cohort and cross-sectional, case-control, before-after, case series, and systematic reviews), from the NHLBI study-quality tools
- CORE extraction, which collects study facts and does not produce a risk-of-bias judgment
QUADAS-2, QUADAS-3, and the CASP checklists are not switched on there. Record those in the evidence table using the columns above. The workspace prepares a draft with quoted passages from the PDF for the profiles that are on; the judgment you export is still yours to confirm. A longer tour of those profiles is in Quality assessment in Evidence Table Builder.
Related reading
Tags:
About the Author
Connect on LinkedInGeorge Burchell
George Burchell is a specialist in systematic literature reviews and scientific evidence synthesis with significant expertise in integrating advanced AI technologies and automation tools into the research process. With over four years of consulting and practical experience, he has developed and led multiple projects focused on accelerating and refining the workflow for systematic reviews within medical and scientific research.