When IPTW Does Not Balance: Revise the Model or the Weights, Then Recheck Overlap and Balance

อ่านฉบับภาษาไทย (Thai version)

Abstract

After inverse probability of treatment weighting (IPTW), the outcome waits until one check has passed: balance, overlap and the distribution of the weights. IPTW does not add patients; it changes how much each one counts, so a treated patient with a propensity score of 0.20 carries a weight of 5 and one with 0.80 carries 1.25. In an illustrative balance table, standardised mean differences (SMDs) of 0.45 to 0.70 before weighting fall to 0.05 or less after it. If an important covariate such as severity still has an SMD of 0.32, the diagnostic has failed: revise the model or weights and recheck overlap and balance. Poor balance with good overlap points to the propensity model, which may need a squared term, spline or interaction. Poor overlap is different: a control patient scored 0.99 gets a weight of 100, and no added term can create comparable patients. This article walks through each check and what to revise when it fails.

---

🔍 Click to enlarge Click anywhere to close

A balance table with one row that will not settle

A research team is comparing Treatment A with no Treatment A in routinely collected hospital data. Physicians gave Treatment A more often to older and more severely ill patients, and more of the treated patients have diabetes. The team estimates propensity scores, weights the patients and opens the balance table.

Age and diabetes now look alike in the two groups, but severity still has a standardised mean difference of 0.32. The outcome analysis is one command away. This article explains why that command should wait, and what to revise first.

The main idea: make the groups comparable

In an observational study, treated and control patients are usually different at baseline. When physicians preferentially give Treatment A to older or more severely ill patients, simply comparing outcomes between the treated and control groups is confounded: the comparison mixes the effect of the treatment with the effect of being older or sicker.

The propensity score (PS) estimates each patient's probability of receiving the treatment given their measured baseline characteristics:

$$PS_i = P(A_i = 1 \mid X_i)$$

Here $A_i = 1$ means that patient $i$ received Treatment A, and $X_i$ represents the patient's measured baseline characteristics. The goal of propensity-score methods is to improve comparability between the treatment groups on these measured baseline covariates [1].

What does IPTW actually do?

Inverse probability of treatment weighting (IPTW) does not literally create new patients. Instead, it changes how much each patient contributes to the analysis.

For the average treatment effect (ATE), the effect of treating everyone compared with treating no one, a simplified form of the weight is

$$w_i = \begin{cases} 1/PS_i, & \text{if treated} \\ 1/(1 - PS_i), & \text{if control} \end{cases}$$

Part 1 of this series builds these weights step by step. Here, two treated patients show what they do.

Two treated patients, two weights

Take two patients who both received the treatment: one for whom it was unusual, and one for whom it was common.

1. A treated patient with PS 0.20

   \[ PS = 0.20 \]

   This patient had only a 20% estimated probability of receiving treatment, yet actually received it.

2. Their weight

   \[ w = 1/0.20 = 5 \]

   Conceptually, this patient contributes approximately five weighted units to the treated pseudo-population, the weighted version of the treated group.

3. A treated patient with PS 0.80

   \[ PS = 0.80 \]

   Receiving treatment was already common among patients with characteristics like theirs.

4. Their weight

   \[ w = 1/0.80 = 1.25 \]

   This patient contributes much less additional weight.

Result: The patient with PS 0.20 counts as about five weighted units and the patient with PS 0.80 as 1.25. Nobody was added to the data; each patient's contribution changed.

The basic workflow

The easiest workflow to remember is

$$\boxed{\text{PS} \rightarrow \text{IPTW} \rightarrow \text{Diagnostics} \rightarrow \text{Outcome}}$$

The word Diagnostics contains two different questions, shown in the table below. These two should not be confused.

Two questions inside 'Diagnostics'

Balance: did IPTW do its job?

Balance is usually judged with the standardised mean difference (SMD): the difference between the two groups' means (or proportions) of a covariate, divided by a pooled standard deviation. Because it is expressed in standard-deviation units, age in years and the proportion with diabetes can be read on one scale.

Suppose that, before weighting, the balance table looks like this.

Before weighting

After IPTW

What happens if the SMD is still 0.32?

Suppose that, after IPTW,

$$\mathrm{SMD}_{\text{severity}} = 0.32$$

This means that the original PS and IPTW specification has not adequately balanced severity. The weighted groups still differ on a covariate that drove the treatment decision, so a comparison of their outcomes would still carry that difference. The analysis should therefore not immediately proceed to the treatment-outcome comparison.

Revising the propensity-score model

Suppose the original model is

$$\operatorname{logit}(PS) = \beta_0 + \beta_1\,\text{Age} + \beta_2\,\text{Severity} + \beta_3\,\text{Diabetes}$$

This assumes, among other things, that a continuous variable such as severity has an adequately modelled relationship with treatment assignment. Reality may be more complicated. Perhaps the probability of treatment changes only slightly from severity 1 to 5, then rises rapidly from severity 6 to 10. A simple linear term, $\beta_2\,\text{Severity}$, may not describe that pattern adequately.

A revised specification could allow nonlinearity, for example

$$\text{Severity} + \text{Severity}^2$$

or use a spline, which lets the effect of severity bend smoothly at a few chosen values.

Another possibility is that treatment assignment depends on combinations of patient characteristics. For example, the interaction

$$\text{Age} \times \text{Severity}$$

could be clinically important if severe disease strongly influences treatment among older patients but not among younger patients.

Austin and Stuart demonstrate this principle by comparing simpler PS specifications with more flexible ones that use restricted cubic splines, and then reassessing the weighted balance [1].

Revising the model does not mean forcing the propensity scores to be equal

Propensity scores are supposed to differ between patients. For example,

$$PS_1 = 0.20,\quad PS_2 = 0.50,\quad PS_3 = 0.85$$

is perfectly normal, because the patients have different baseline characteristics. The goal is not

$$PS_1 = PS_2 = PS_3$$

The goal is that, after the weights are applied, the distribution of important baseline characteristics becomes similar between the treated and control groups.

What is overlap?

Now consider a different problem. Imagine that the propensity scores of the control group run from 0.05 to 0.40 and those of the treated group from 0.30 to 0.90:

Control:  0.05 ───────── 0.40
Treated:            0.30 ───────── 0.90

There is a region, approximately from 0.30 to 0.40, where both treated and control patients exist. That is overlap, or common support. Within it we can find patients with reasonably similar baseline probabilities of treatment who happened to receive different treatments.

Now imagine that the control scores run from 0.02 to 0.20 and the treated scores from 0.80 to 0.99:

Control:  0.02 ─── 0.20

Treated:                    0.80 ─── 0.99

There is almost no overlap. The data are telling us that patients with one type of baseline profile almost always received control, while patients with another type almost always received treatment. There are very few comparable counterfactual patients: for a typical treated patient, there is no control patient with a similar profile to show what would have happened without treatment.

This is related to the positivity assumption: every type of patient being compared must have some real chance of receiving each treatment.

Why poor overlap causes extreme weights

Suppose a control patient has a propensity score of 0.99.

1. The weight of a control patient

   \[ w = \frac{1}{1 - PS} \]

   A control patient is weighted by the inverse of their probability of receiving control.

2. Put in the score

   \[ w = \frac{1}{1 - 0.99} = 100 \]

   One observation receives an enormous weight.

Result: A single control patient with PS 0.99 gets a weight of 100. The model is essentially saying: someone with these characteristics almost always receives treatment, but this unusual person received control.

What one weight of 100 does

That patient carries huge statistical influence, which may produce unstable estimates and large variance. Examining the distribution of the weights, particularly unusually extreme weights, is therefore part of IPTW diagnostics. It can also give clues about positivity problems or about a misspecified PS model. Part 3 of this series looks at extreme weights in more detail.

Poor balance and poor overlap are not the same problem

Why polynomial terms cannot create true overlap

Suppose almost every extremely sick patient receives treatment, and almost every mild patient receives control:

$$P(\text{Treatment} = 1 \mid \text{severe}) = 0.99, \quad P(\text{Treatment} = 1 \mid \text{mild}) = 0.05$$

There may simply be no severe control patients available for a meaningful comparison. Adding $\text{Severity}^2$ or $\text{Severity}^3$ to the PS model cannot magically create severe control patients.

“Revise the weights” is not “revise the PS model”

The two halves of “revise the model or weights” are different actions, and they should be kept apart.

Revise the PS model

Here you are changing how treatment assignment is modelled. For example,

$$\text{Age} + \text{Severity}$$

might become

$$\text{Age} + \text{Severity} + \text{Severity}^2 + \text{Age} \times \text{Severity}$$

Then you calculate a new PS and new weights. The objective is better covariate balance.

Revise the weighting strategy

Here the concern may be extreme weights or limited overlap. Depending on the research question and the estimand, possible strategies include stabilised weights (weights multiplied by the overall share of patients receiving that treatment, which narrows their spread), carefully justified trimming (removing patients with extreme scores) or truncation (capping the largest weights), or a different estimand, such as the one targeted by overlap weighting.

These approaches are not interchangeable, because they can change which population the estimated treatment effect represents. Part 3 of this series sets out what each one changes.

A small example: ten patients

Imagine ten patients. Before weighting:

Treated:
mostly old + severe patients

Control:
mostly young + mild patients

But there are a few unusual patients:

young patient → received treatment
old severe patient → received control

These unusual patients are very informative, because they provide the kind of treatment contrast that is otherwise uncommon in the data. IPTW therefore gives them greater statistical weight. After weighting, we might conceptually obtain:

Weighted Treated:
old old old + young young young young

Weighted Control:
young young young + old old old old

Again, no patients were cloned in reality. This is a pseudo-population created mathematically through weights.

The correct mental model

Do not memorise a long sequence such as

$$\text{PS} \rightarrow \text{weights} \rightarrow \text{overlap} \rightarrow \text{SMD} \rightarrow \text{revise} \rightarrow \cdots$$

Instead, remember one short chain with a single check in the middle.

Back to the severity SMD of 0.32

Return to the team at the start. After IPTW, an important covariate, severity, still has a standardised mean difference of 0.32. So we know:

$$\text{PS} \rightarrow \text{IPTW} \rightarrow \boxed{\mathrm{SMD} = 0.32}$$

The diagnostic has failed, so the team should not yet interpret the outcome. Instead, they should revise the model or weights and recheck overlap and balance.

If comparable patients exist across the range of severity, the first step is a more flexible PS model: a squared term or a spline for severity, or an age-by-severity interaction, followed by new weights and a new balance table. If almost all the severely ill patients received Treatment A, no added term will help, and the decision becomes one about the weighting strategy and the estimand.

Memory map

Common misreadings and their fixes

• “IPTW creates extra patients.”

  No patient is added or cloned. A patient with a weight of 5 is one person who counts as about five weighted units.

  Fix: Read each weight as that patient's statistical representation in the pseudo-population.

• “Once the weights are made, the outcome can be analysed.”

  Weighting only creates the synthetic population. Whether it is comparable is not yet known.

  Fix: Run the CHECK first: balance, overlap and the distribution of the weights. Troubleshoot before any outcome analysis if it fails.

• “A P value tells me whether the groups are balanced.”

  Balance is judged with standardised differences, not significance tests [1], and the 0.10 cut-off is a convention, not a biological threshold.

  Fix: Read the SMD of every important covariate after weighting, and treat 0.10 as a rule of thumb.

• “Revising the PS model means making the scores more alike.”

  Scores of 0.20, 0.50 and 0.85 are normal, because the patients differ at baseline.

  Fix: Judge a revision by whether the weighted distributions of the important baseline characteristics become similar.

• “Keep adding terms until the model looks right.”

  Revision is an iterative diagnostic process driven by residual imbalance, not automatic model fishing.

  Fix: Reconsider the specification on clinical grounds, then refit, reweight and check balance again.

• “A squared term or an interaction will fix poor overlap.”

  When almost every severe patient was treated, Severity² cannot create severe control patients.

  Fix: Look at the overlap of the scores; if comparable patients are absent, reconsider the target population or the weighting strategy.

• “Stabilised weights, trimming and overlap weighting are interchangeable.”

  They can change which population the estimated treatment effect represents.

  Fix: Choose the weighting strategy from the research question and the estimand, and state which population the estimate describes.

Glossary

propensity score (คะแนนแนวโน้มการได้รับการรักษา)

The probability of receiving treatment given the measured baseline characteristics.

inverse probability of treatment weighting (การถ่วงน้ำหนักด้วยส่วนกลับของความน่าจะเป็นในการได้รับการรักษา (IPTW))

Weighting each patient by the inverse probability of the treatment they received, which changes how much each patient counts.

pseudo-population (ประชากรเสมือน)

The weighted, synthetic population that IPTW creates mathematically; nobody is added or cloned.

standardised mean difference (ผลต่างค่าเฉลี่ยมาตรฐาน)

The difference in a covariate's mean or proportion between groups, divided by a pooled standard deviation.

overlap (common support)

The range of propensity scores in which both treated and control patients exist.

positivity (ข้อสมมติ positivity)

Every type of patient being compared has some real chance of receiving each treatment.

overlap weighting (น้ำหนักแบบทับซ้อน (overlap weight))

Weighting that emphasises patients where the treated and control groups overlap and down-weights the tails, which changes the estimand.

estimand (ปริมาณเป้าหมายของการประมาณ)

The effect a study aims to estimate, including the population it refers to.

stabilised weight (น้ำหนักแบบปรับเสถียร)

An inverse probability weight multiplied by the overall share of patients receiving that treatment, which narrows the spread of the weights.

trimming (การตัดผู้ป่วยออกจากการวิเคราะห์)

Removing patients with extreme propensity scores, which changes the population the estimate describes.

truncation (การตัดปลายน้ำหนัก)

Capping the largest weights at a chosen value.

References

1. Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med. 2015;34(28):3661-3679. doi:10.1002/sim.6607 https://doi.org/10.1002/sim.6607
2. Li F, Morgan KL, Zaslavsky AM. Balancing covariates via propensity score weighting. J Am Stat Assoc. 2018;113(521):390-400. doi:10.1080/01621459.2016.1260466 https://doi.org/10.1080/01621459.2016.1260466

Key takeaways

• IPTW does not add patients; it changes how much each one counts, so a treated patient with PS 0.20 gets a weight of 5 and one with PS 0.80 gets 1.25.
• After weighting, CHECK before the outcome: balance (usually the SMD), overlap and the distribution of the weights.
• An important covariate with an SMD of 0.32 after IPTW means the diagnostic has failed: revise the model or weights and recheck overlap and balance.
• With good overlap, residual imbalance points to the PS model (squares, splines, plausible interactions); different propensity scores are normal.
• If overlap is truly absent, no added term can create comparable patients; that calls for a decision on the weighting strategy and the estimand.

Related in the wiki: [[iptw-guide]]