Beyond the Evidence Pyramid: GRADE Certainty and Choosing Effectiveness Inputs for Economic Models

อ่านฉบับภาษาไทย (Thai version)

Abstract

A district hospital committee must put one number for the effect of C into its economic model. Its best evidence is a randomized trial: 8 of 66 patients on C and 13 of 60 on usual care, A, had a serious complication within a year, a risk ratio of 0.56 with a 95% confidence interval from 0.25 to 1.26. An evidence pyramid ranks the trial near the top. GRADE asks how sure we can be about this effect, for this outcome. The trial starts at high certainty and drops two levels to low. Its interval runs from three quarters fewer complications to a quarter more, so it cannot separate a worthwhile benefit from a harm. It also has 126 of the 1,638 patients needed to show a quarter fewer. Low certainty is not zero effect. On the district's assumed risk of 22.0%, 0.56 means 9.7 complications avoided per 100 patients; the model must show results from 16.5 avoided to 5.6 more.

🔍 Click to enlarge Click anywhere to close

One number for the model, and a pyramid that answers too fast

A district hospital committee must decide whether to pay for C, a new preventive treatment for adults newly diagnosed with a chronic condition. Its economic model needs one number for C: how much it changes the risk of a serious complication within one year, compared with usual care, A. The district diagnoses about 100 such patients a year.

The best evidence on the desk is a randomized trial. Of 66 patients given C, 8 had a serious complication within a year; of 60 given A, 13 did. These numbers are illustrative, for teaching, not results of a real study. A second source is on the desk too: the records of a provincial hospital, where doctors chose who received C, which this article uses to show what the pyramid cannot see.

That is 12.12% against 21.67%, or 9.55 fewer complications per 100 patients on C. As a risk ratio it is 0.56, with a 95% confidence interval from 0.25 to 1.26 (Risk Ratio, Odds Ratio, Risk Difference and NNT: One 2×2 Table, Four Answers). Before rounding these are 0.559, 0.249 and 1.256, the values the model uses later.

One member points to an evidence pyramid, a chart that ranks study designs: systematic reviews at the top, randomized trials in the second tier, then cohort studies, then case-control studies. Case-control studies choose patients by whether they had the outcome, so what they estimate depends on how the comparison patients were sampled (Design, Model and Estimand: What Each Study Design Can Actually Estimate). Below them come case series, reports on a group of patients with no comparison group, and expert opinion sits at the bottom. The trial sits in that second tier, so this member wants 0.56 in the model as it is.

Another member brings a second pyramid from a different handout. It puts a review that pools observational studies below a single trial, while the first pyramid puts every review at the top.

When two charts rank the same designs differently, ranking designs is probably the wrong question. The committee's real question is how sure it can be that C lowers complications by about as much as 0.56 says. This article answers it for this trial, then shows what the model does with the answer.

What a pyramid ranks, and what it cannot see

Read first on this site:

• How systematic reviews and meta-analyses pool studies into one answer
• How to judge whether a systematic review was done well
• Which study designs answer questions of harm, diagnosis and prognosis

A pyramid ranks designs by how prone they usually are to bias: a push on the result in one direction that has nothing to do with the treatment and that more patients do not remove. A randomized trial earns its high place because chance alone decides who gets C. Patients on C and on A then differ before treatment only by chance, not because a doctor chose.

In the provincial hospital's records, 500 patients got C and 500 got A, and doctors gave C mostly to severely ill patients. Of the 500 on C, 400 were severe (120 had a complication, 30.0%) and 100 mild (3, 3.0%). Of the 500 on A, 100 were severe (40, 40.0%) and 400 mild (16, 4.0%).

Pooled, that is 123 of 500 on C (24.6%) against 56 of 500 on A (11.2%), a ratio of 2.20, more than twice the complications on C. Severity, not C, made that difference; mixing another cause of the outcome into the comparison is called confounding (Chance, Bias and Confounding: Why an Association Is Not Yet a Treatment Effect). Comparing severe with severe and mild with mild, then giving each group half the weight, gives 16.5% on C against 22.0% on A, a ratio of 0.75 with an interval from 0.57 to 0.99.

This is also why the top of the first pyramid can mislead. A review that pooled several record studies like this one, each comparing all patients with severity ignored, would return a narrower interval around the same 2.20. Pooling shrinks chance, not confounding or other bias, so such a review gives a precise estimate of a biased effect.

But a pyramid looks only at the design label. It cannot see that only 21 of the trial's 126 patients had a complication, which is why the interval is wide. Nor can it see whether the staff who counted complications knew who got C, or whether the trial's patients resemble the district's.

So the committee needs a judgement about the evidence itself, for this one outcome. All the studies that answer the question for that outcome are called the body of evidence; here it is one trial. Certainty is how confident we can be that the true effect lies close to the estimate.

GRADE: start from the design, then look for reasons to move

Read first on this site:

• The GRADE framework in full: the four levels and every reason to move between them
• How to appraise a systematic review and the certainty of its results

GRADE, short for Grading of Recommendations Assessment, Development and Evaluation, is a shared method for rating that certainty [1]. The linked post teaches the full method; here we need only the verdict. GRADE ends at one of four levels, and the model carries that level beside each input:

• High: the true effect is very likely close to the estimate.
• Moderate: it is probably close, but could be substantially different.
• Low: it may be substantially different.
• Very low: it is probably substantially different.

The rating is given per outcome, so one trial can earn high certainty for deaths and low certainty for complications. The model carries the rating of the outcome it uses, here serious complications within a year.

The rating starts from the design. Randomized trials start at high. Observational studies, such as the provincial records, start at low, because some confounding usually survives even after comparing within severity groups. The design sets only this starting point: a trial can still end at low, and observational evidence can move up.

Then five kinds of problem can move the rating down: one level if serious, two if very serious. Below, each is checked on the trial.

Five reasons to rate down, checked on the trial

Read first on this site:

• A pocket guide to appraising a randomized trial
• Judging risk of bias in a study without randomization, domain by domain
• Robust ways to pool studies, including when their results differ

Risk of bias asks whether flaws in how the study was run could push its result one way, judged with RoB 2 for randomized trials [2] and ROBINS-I for studies without randomization [3]. Here nobody could foresee the next patient's treatment, and complications came from hospital admission records: not serious.

Inconsistency asks whether several studies of the same question disagree more than chance explains. With one trial there is nothing to disagree with: not serious. The records' 0.75 happens to lie inside the trial's interval of 0.25 to 1.26, but the records are a different body of evidence and play no part in this rating.

When several trials are pooled, the pooled interval says where the average effect lies. A single new setting such as the district can land elsewhere, especially if the trials disagree. The range of effects it may see is the prediction interval, and it is the one the model needs (Robust Approaches for Conventional Meta-Analysis and Network Meta-Analysis).

Indirectness asks whether the evidence answers a different question: other patients, another comparison, or another outcome such as a surrogate outcome, a blood marker that stands in for complications and counts only if trials have shown that lowering it prevents them. Otherwise the gain the model gives C would rest on an assumption; our trial enrolled the same kind of patients in the same district, compared C with A and counted complications themselves: not serious.

Imprecision asks whether the interval is too wide to choose between the options. The trial's interval runs from 0.25 to 1.26; the next section shows why this is the trial's one very serious problem.

Publication bias asks whether studies are missing because their results disappointed, which a review checks by searching trial registries, where trials are listed before they start. None was found here: not suspected.

Imprecision: is the interval narrow enough to decide?

An interval is too wide when the decision would change depending on where in it the truth lies. So the committee first states its decision boundaries: the smallest benefit and the smallest harm it would act on.

It calls a quarter fewer complications, a risk ratio of 0.75, an important benefit. A quarter more, 1.25, is an important harm. These boundaries are the committee's illustrative choice; GRADE asks each panel to state its own.

The trial's interval reaches past both. Its lower end, 0.25, would mean 75% fewer complications on C; its upper end, 1.26, would mean 26% more. Read against the boundaries, the data cannot tell an important benefit from an important harm. The upper end only just passes 1.25, so the second check below matters.

That check asks how far the trial falls short of enough patients. A trial is planned to be big enough that, if C truly cut complications by a quarter, 80 of every 100 such trials would give an interval ending below 1. That 80% is called the trial's power: its chance of showing a benefit that is really there.

The name and the formula

The number of patients a single trial needs to detect the important benefit is called the optimal information size. It is GRADE's second check on imprecision: evidence from fewer patients counts as imprecise even when its interval looks acceptable.

With $p_A$ and $p_C$ the planned risks on A and on C, and $z_{0.975}=1.96$ and $z_{0.80}=0.8416$ the two multipliers, the steps above are one formula:

$$n=\frac{(z_{0.975}+z_{0.80})^2\,[p_A(1-p_A)+p_C(1-p_C)]}{(p_A-p_C)^2}$$

What would a trial of that size show? Ratios are easier to widen and narrow on the log scale, where equal steps mean equal multiplications. There the trial's ratio before rounding, 0.5594, becomes $\ln 0.5594=-0.581$, and its interval is -0.581 plus or minus 1.96 times a spread.

The spread says how much a ratio built from so few events can wobble by chance. The two event counts, 8 and 13, add most of it, and the group sizes, 66 and 60, take a little back: $\sqrt{1/8-1/66+1/13-1/60}=\sqrt{0.1701}=0.4124$, derived in the linked post (Risk Ratio, Odds Ratio, Risk Difference and NNT: One 2×2 Table, Four Answers).

The spread shrinks with the square root of the number of patients, so four times as many patients halve it. With 1,638 patients instead of 126 patients, the spread becomes $0.4124\times\sqrt{126/1638}=0.4124\times0.2774=0.1144$. If the true ratio were the trial's 0.56, the interval would be $e^{-0.581\pm1.96\times0.1144}=e^{-0.581\pm0.224}$, from 0.45 to 0.70, clear of both boundaries. The code boxes below trace the trial, the information size and this narrower interval.

# Illustrative numbers for the committee's case: the randomized trial of C against usual care A.
risk_c <- 8 / 66; risk_a <- 13 / 60; ratio <- risk_c / risk_a
spread <- sqrt(1 / 8 - 1 / 66 + 1 / 13 - 1 / 60)
limits <- exp(log(ratio) + c(-1.96, 1.96) * spread)
f <- function(x, d) formatC(x, d, format = "f")
cat("Risk on C:", f(100 * risk_c, 2), "%  on A:", f(100 * risk_a, 2), "%\n")
cat("Risk ratio:", f(ratio, 2), "  (", f(ratio, 3), ")\n")
cat("95% interval:", f(limits, 2), "  (", f(limits, 3), ")\n")
cat("Lower limit, a reduction of:", f(100 * (1 - limits[1]), 0), "%\n")
cat("Upper limit, an increase of:", f(100 * (limits[2] - 1), 0), "%\n")
cat("Crosses the boundaries 0.75 and 1.25:", limits[1] < 0.75, limits[2] > 1.25, "\n")
cat("Fewer complications per 100 on C:", f(100 * (risk_a - risk_c), 2), "\n")
cat("Patients:", 66 + 60, "  with a complication:", 8 + 13, "\n")

Risk on C: 12.12 %  on A: 21.67 %
Risk ratio: 0.56   ( 0.559 )
95% interval: 0.25 1.26   ( 0.249 1.256 )
Lower limit, a reduction of: 75 %
Upper limit, an increase of: 26 %
Crosses the boundaries 0.75 and 1.25: TRUE TRUE
Fewer complications per 100 on C: 9.55
Patients: 126   with a complication: 21

# Illustrative numbers for the committee's case: patients a trial needs to show a quarter fewer.
risk_a <- 13 / 60; risk_c <- risk_a * 0.75; gap <- risk_a - risk_c
noise <- risk_a * (1 - risk_a) + risk_c * (1 - risk_c)
z <- c(qnorm(0.975), qnorm(0.80)); per_arm <- ceiling(sum(z)^2 * noise / gap^2)
f <- function(x, d) formatC(x, d, format = "f")
cat("Risks on A and C:", f(c(risk_a, risk_c), 5), "  gap:", f(gap, 5), "\n")
cat("p(1 - p) on A and C:", f(c(risk_a * (1 - risk_a), risk_c * (1 - risk_c)), 4), "\n")
cat("Their sum:", f(noise, 4), "  gap squared:", f(gap^2, 6), "\n")
cat("Multipliers:", f(z, 4), "  sum:", f(sum(z), 4), "  squared:", f(sum(z)^2, 3), "\n")
cat("Per arm:", f(sum(z)^2 * noise / gap^2, 1), "-> up to", per_arm, "  total:", 2 * per_arm, "\n")
cat("Trial's share:", f(100 * 126 / (2 * per_arm), 1), "%\n")

Risks on A and C: 0.21667 0.16250   gap: 0.05417
p(1 - p) on A and C: 0.1697 0.1361
Their sum: 0.3058   gap squared: 0.002934
Multipliers: 1.9600 0.8416   sum: 2.8016   squared: 7.849
Per arm: 818.1 -> up to 819   total: 1638
Trial's share: 7.7 %

# Illustrative numbers for the committee's case: the trial's risks with 1,638 patients, not 126.
ratio <- (8 / 66) / (13 / 60); spread <- sqrt(1 / 8 - 1 / 66 + 1 / 13 - 1 / 60)
shrink <- sqrt(126 / 1638); new_spread <- spread * shrink
f <- function(x, d) formatC(x, d, format = "f")
cat("Ratio before rounding:", f(ratio, 4), "  its log:", f(log(ratio), 3), "\n")
cat("Spread squared:", f(spread^2, 4), "  spread:", f(spread, 4), "\n")
cat("Shrink factor:", f(shrink, 4), "  new spread:", f(new_spread, 4), "\n")
cat("Half-width, 1.96 x new spread:", f(1.96 * new_spread, 3), "\n")
cat("95% interval with 1,638 patients:", f(exp(log(ratio) + c(-1.96, 1.96) * new_spread), 2), "\n")

Ratio before rounding: 0.5594   its log: -0.581
Spread squared: 0.1701   spread: 0.4124
Shrink factor: 0.2774   new spread: 0.1144
Half-width, 1.96 x new spread: 0.224
95% interval with 1,638 patients: 0.45 0.70

Reasons to rate up, and why the records stay low

Observational evidence can also move up, for three reasons [1]. Each names a pattern that bias alone would struggle to produce, and each is usually considered only when nothing has been rated down.

A large effect: a ratio below 0.5, or above 2, is hard to explain by confounding alone. The records' 0.75 is not one.

A dose-response gradient: complications fall step by step as patients receive more C, or receive it for longer. The records hold no doses, so there is nothing to see.

Confounding that works against the result: if every plausible leftover confounder would push the estimate toward no effect, a benefit that still shows is more believable. In the records it could run either way, hiding benefit or inflating it, so it does not apply.

None of the three reasons applies to the records, so they start at low and cannot rise above it.

Some reviewers instead start non-randomized studies at high and let ROBINS-I [3], which compares a study with the ideal trial it should have imitated, do the lowering; either route ends at low for the records, and each worry is counted once.

So neither number on the committee's desk carries more than low certainty. The pyramid hid this: it ranked the trial one tier above the records without looking at either study's evidence.

# Illustrative numbers for the committee's case: the provincial records, every patient pooled.
n_c <- c(severe = 400, mild = 100); e_c <- c(severe = 120, mild = 3)
n_a <- c(severe = 100, mild = 400); e_a <- c(severe = 40, mild = 16)
pool_c <- sum(e_c) / sum(n_c); pool_a <- sum(e_a) / sum(n_a)
f <- function(x, d) formatC(x, d, format = "f")
cat("Patients on C:", sum(n_c), "  severe:", n_c[[1]], "  mild:", n_c[[2]], "\n")
cat("Patients on A:", sum(n_a), "  severe:", n_a[[1]], "  mild:", n_a[[2]], "\n")
cat("Complications on C, severe and mild:", e_c, "  risks:", f(100 * e_c / n_c, 1), "%\n")
cat("Complications on A, severe and mild:", e_a, "  risks:", f(100 * e_a / n_a, 1), "%\n")
cat("Pooled on C:", sum(e_c), "of", sum(n_c), "  risk:", f(100 * pool_c, 1), "%\n")
cat("Pooled on A:", sum(e_a), "of", sum(n_a), "  risk:", f(100 * pool_a, 1), "%\n")
cat("Pooled ratio, severity ignored:", f(pool_c / pool_a, 2), "\n")

Patients on C: 500   severe: 400   mild: 100
Patients on A: 500   severe: 100   mild: 400
Complications on C, severe and mild: 120 3   risks: 30.0 3.0 %
Complications on A, severe and mild: 40 16   risks: 40.0 4.0 %
Pooled on C: 123 of 500   risk: 24.6 %
Pooled on A: 56 of 500   risk: 11.2 %
Pooled ratio, severity ignored: 2.20

# Illustrative numbers for the committee's case: the provincial records, half the patients severe.
share <- c(severe = 0.5, mild = 0.5)
risk_c <- c(120 / 400, 3 / 100); patients_c <- c(400, 100)
risk_a <- c(40 / 100, 16 / 400); patients_a <- c(100, 400)
std_c <- sum(share * risk_c); std_a <- sum(share * risk_a); ratio <- std_c / std_a
spread <- sqrt(sum(share^2 * risk_c * (1 - risk_c) / patients_c) / std_c^2 +
               sum(share^2 * risk_a * (1 - risk_a) / patients_a) / std_a^2)
f <- function(x, d) formatC(x, d, format = "f")
cat("Severe and mild, risk on C:", f(100 * risk_c, 1), "%  on A:", f(100 * risk_a, 1), "%\n")
cat("All patients pooled, severity ignored:", f((123 / 500) / (56 / 500), 2), "\n")
cat("Same patients on C and on A:", f(100 * c(std_c, std_a), 1), "%  ratio:", f(ratio, 2), "\n")
cat("95% interval:", f(exp(log(ratio) + c(-1.96, 1.96) * spread), 2), "\n")
cat("Large effect, a ratio below 0.5?", ratio < 0.5, "\n")

Severe and mild, risk on C: 30.0 3.0 %  on A: 40.0 4.0 %
All patients pooled, severity ignored: 2.20
Same patients on C and on A: 16.5 22.0 %  ratio: 0.75
95% interval: 0.57 0.99
Large effect, a ratio below 0.5? FALSE

From a ratio to complications avoided in the district

The model counts complications, not ratios. So the committee multiplies the ratio by the district's baseline risk on A: the risk its patients would have without C.

Why the ratio and not the trial's difference? In the provincial records C gave a ratio of 0.75 in severe patients, 30.0% on C against 40.0% on A, and the same 0.75 in mild patients, 3.0% against 4.0%. The difference, though, was 10 points in one group and 1 point in the other. A ratio tends to carry from one group of patients to another; a difference does not.

The committee has no count of its own, so it assumes the district's patients have the records' mix: half severe, with those risks on A of 40.0% and 4.0%. That gives $0.5\times40.0\%+0.5\times4.0\%=22.0\%$. On C the risk becomes $22.0\%\times0.559=12.3\%$, using the trial's unrounded ratio, so C avoids 9.7 complications per 100 patients, about 9.7 a year in this district.

Using the trial's own difference, 9.55 fewer per 100 patients, would give almost the same answer here, because the trial's risk on A, 21.67%, is close to 22.0%. It would mislead elsewhere. In a clinic whose patients were all mild, with a risk on A of 4.0%, the same ratio avoids only 1.8 complications per 100 patients.

Whether an effect measured in one group of patients holds in another is called transportability. The usual approach takes the relative effect, here the ratio, from the best evidence and the baseline risk from local data [4], here assumed from the records' mix because the district has no count of its own. It then checks that nothing about the local patients, such as severity or how C is given, would change the ratio itself.

Low certainty becomes scenarios

Read first on this site:

• Uncertainty and sensitivity analysis in a health economic evaluation

Low certainty does not mean the effect is zero. It means the committee should see its decision under each effect the evidence still allows.

Running the model at the interval's ends, at the trial's estimate and at the other values on the desk gives the table below, with the unrounded ratios. At the lower end, 0.249, C avoids 16.5 complications per 100 patients. At the upper end, 1.256, it causes 5.6 more. The records' 0.75 gives 5.5 avoided, and no effect gives 0.0.

# Illustrative numbers for the committee's case: the trial's ratio on the district's risk on A.
ratio <- (8 / 66) / (13 / 60); spread <- sqrt(1 / 8 - 1 / 66 + 1 / 13 - 1 / 60)
ratios <- c(ratio * exp(-1.96 * spread), ratio, 0.75, 1, ratio * exp(1.96 * spread))
risk_a <- 0.5 * 0.40 + 0.5 * 0.04   # half the patients severe, as in the provincial records
avoided <- 100 * risk_a * (1 - ratios)
f <- function(x, d) formatC(x, d, format = "f")
cat("District's risk on A:", f(100 * risk_a, 1), "%\n")
cat("Ratios tried:", f(ratios, 3), "\n")
cat("Risk on C:", f(100 * risk_a * ratios, 1), "%\n")
cat("Avoided per 100 patients:", f(avoided, 1), "\n")
cat("Trial's own difference:", f(100 * (13 / 60 - 8 / 66), 2), "  mild only (risk on A 4.0%):",
    f(100 * 0.04 * (1 - ratio), 1), "\n")

District's risk on A: 22.0 %
Ratios tried: 0.249 0.559 0.750 1.000 1.256
Risk on C: 5.5 12.3 16.5 22.0 27.6 %
Avoided per 100 patients: 16.5 9.7 5.5 0.0 -5.6
Trial's own difference: 9.55   mild only (risk on A 4.0%): 1.8

What the interval covers, and what else the report needs

Re-running the model across the trial's interval covers chance: the uncertainty that a bigger trial would shrink [4]. It does not cover a bias the trial might carry, or a difference between the trial's patients and the district's. Each of those needs its own scenario, stated and tested one at a time.

Here the model compares only C with A. When it must compare several treatments, the effects often come from a network meta-analysis, a review that links trials through the treatments they share (Network Meta-Analysis Explained: Comparing Multiple Treatments in One Framework). Each comparison then gets its own certainty rating, and a ranking of the treatments is never read on its own.

When the model is reported, the CHEERS 2022 checklist for reporting economic evaluations asks where each effect estimate came from [5]. Writing its certainty beside it, here low because of imprecision, shows readers which inputs the decision leans on.

Whether a bigger trial is worth paying for is a calculation of its own, the value of more information. It asks how often the new result would change the committee's choice, and what a wrong choice would cost [6]. The series returns to it once the model is built.

The series, in learning order

1. Health Economics Series Guide: Read the Evidence Before the Costs
2. Incidence, Prevalence and the Right Denominator: The Epidemiology Beneath Cost-of-Illness and Markov Models
3. Design, Model and Estimand: What Each Study Design Can Actually Estimate
4. Risk Ratio, Odds Ratio, Risk Difference and NNT: One 2×2 Table, Four Answers
5. The Hazard Ratio Is Not a Risk Ratio: Proportional Hazards, Restricted Mean Survival Time and Model Inputs
6. Chance, Bias and Confounding: Why an Association Is Not Yet a Treatment Effect
7. Beyond the Evidence Pyramid: GRADE Certainty and Choosing Effectiveness Inputs for Economic Models (this article)
8. From Sensitivity to Cost per Case Detected: Predictive Values, Prevalence and the Economics of Screening
9. Whose Cost, in Which Year? Perspective, Opportunity Cost, Inflation and Discounting
10. QALY, DALY and the Utilities Behind Them: What Is Actually Being Measured
11. ICER, Dominance and the Cost-Effectiveness Plane: Reading the Thai Threshold of 160,000 Baht per QALY
12. Decision Trees, Markov Cohorts and Probabilistic Sensitivity Analysis: How a Model Turns Evidence into a Decision
13. Cost-Effective Is Not Affordable, and Efficient Is Not Fair: Budget Impact and Equity
14. Parameter Sourcing, Reference Cost Lists and Utility Catalogues When Local Data Are Sparse
15. Early Health Technology Assessment, Headroom Pricing and Benefit-Package Decisions
16. Communicating Health Economic Evidence: From Model Outputs to Policy Briefs

Related in the wiki: Risk Ratio, Odds Ratio, Risk Difference and NNT: One 2×2 Table, Four Answers · Chance, Bias and Confounding: Why an Association Is Not Yet a Treatment Effect · From Sensitivity to Cost per Case Detected: Predictive Values, Prevalence and the Economics of Screening · Decision Trees, Markov Cohorts and Probabilistic Sensitivity Analysis: How a Model Turns Evidence into a Decision · Design, Model and Estimand: What Each Study Design Can Actually Estimate · GRADE Framework: A Systematic Approach to Rating the Quality of Evidence