Rigour Isn't the Problem.

There’s a familiar moment in many systematic reviews. You’re deep into quality assessment, using a framework you’ve worked with before. You know the domains, the signalling questions, and you understand the logic behind them.

And yet you pause. You reread the same question over and over again, to then ask yourself whether your judgement is “defensible enough”.

This moment came into focus while I was helping develop a quality assessment tool for systematic reviews, working directly with established frameworks like RoB 2, ROBINS-I, and ROBIS. All methodologically sound tools that are widely accepted. They are built by people who know the science inside out. However, people still hesitate.

---

The Real Situation Researchers Are Working In

The challenge was about how frameworks show up in real research conditions. I’m thinking, mid-review, late in the day, when someone is tired, under pressure, and trying to stay consistent across dozens of papers. That’s the environment most quality assessment actually happens in.

When you put these tools into that context, something becomes obvious very quickly: most difficulty isn’t about misunderstanding the questions. It’s about uncertainty.

The questions can be technically precise and still feel psychologically ambiguous. Reviewers know what the framework says, but they’re unsure how their judgement will land. They worry about edge cases or whether someone else would score it differently. They also have concerns about whether they’ll have to justify themselves later.

The result of this is that their work becomes slower due to overthinking and second-guessing. And none of that shows up in the final table.

---

Rigour Without Reassurance Creates Friction

This experience reinforced a belief I now hold quite strongly: rigour without reassurance creates friction.

Frameworks like RoB 2 and ROBIS assume a level of confidence, shared interpretation, and stability that isn’t always present in day-to-day workflows. When that support isn’t built into the tool or the process, people compensate in predictable ways. They become inconsistent and doubtful of themselves.

Often, the response to this is “they just need more training”. Sometimes that is true, but not always. In many cases, the issue is a lack of support for human judgement under pressure. The frameworks are doing exactly what they were designed to do—but they weren’t designed with cognitive load, fatigue, or decision anxiety in mind. That gap matters more than we tend to admit.

---

Quality Assessment Is a Decision-Making Environment

One shift this work caused for me was how I think about quality assessment altogether.

I stopped seeing it as a purely methodological exercise and started seeing it as a decision-making environment. The work isn’t just encoding domains correctly, it also involves helping people make judgments they feel able to stand behind.

That means reducing unnecessary ambiguity. Clarifying what “yes”, “probably yes”, or “some concerns” actually look like in practice, which makes reasoning explicit rather than assumed.

When someone records a judgement, they shouldn’t feel like they’re guessing how it will be perceived later. They should feel confident that their decision is reasoned, explainable, and consistent with the framework’s intent.

---

Where Self-Doubt Creeps In

This resonated with me because I’ve seen the same pattern repeatedly.

Brilliant researchers lose confidence because the tools ask them to make high-stakes judgements without enough structural support. Quality assessment is one of those stages where self-doubt creeps in quietly. There’s no obvious error message, but just a growing sense of “I think this is right… but I’m not entirely sure”.

Once that feeling sets in, everything seems to slow down. That means that decisions take longer, consistency drops, and reviews drag on because they care a lot and don’t want to get it wrong. This highlights a design problem.

---

The Question That Stays With Me

The work left me with a question I now ask often, especially when I see hesitation or inconsistency in quality assessment:

When we notice these issues, do we assume the reviewer needs more training—or do we ask whether the tool is actually designed to support human judgement under real conditions?

That question has changed how I think about methodology, software, and even leadership, because the goal is helping careful people make careful decisions with confidence.

When we design tools and processes that acknowledge how people actually work (tired, thoughtful, cautious, under pressure) we don’t weaken rigour. We should protect it. And in the long run, that’s what leads to better, more trustworthy research.