Skip to content

BUG: Raise for nan in cluster ids - #845

Merged
bashtage merged 3 commits into
mainfrom
lm71-bug8-panel-nan-clusters
Oct 15, 2026
Merged

bashtage merged 3 commits into
mainfrom
lm71-bug8-panel-nan-clusters

Conversation

@bashtage

Copy link
Copy Markdown
Owner

#789 made the IV and system models raise a ValueError when the cluster ids have
missing values. The panel models did not. They replace the cluster ids by the codes
of their categories before the covariance is computed, and the code of a missing
value is -1, so the observations with a missing id became a cluster of their own. In
a PooledOLS example with 120 observations and 7 missing ids, the standard error of
the constant changed from 0.0855 to 0.0900 without an error. BetweenOLS and
FirstDifferenceOLS, which need the clusters to be constant within an entity, failed
in other ways: BetweenOLS ignored a missing number if another observation of the
entity had an id, FirstDifferenceOLS ignored a missing id in the first period of an
entity and otherwise raised the error of pandas about converting NaN to an integer,
and text and categories gave "clusters must not vary within an entity" or "clusters
must be identical for values used to compute the first difference".

reformat_clusters, which every panel model uses to place the clusters on the rows of
dependent, now raises the ValueError of the IV and system models, "clusters cannot
contain missing values.", before text and categories are replaced by codes. This
covers every form of clusters: a pandas object or PanelData with the index of
dependent, a 1-d array or list, a grid (a 2-d or 3-d array, a DataFrame of the time
periods by the entities, or a DataArray), and text and categories, with one or two
variables.

Only the observations that the model keeps are checked. The ids of the observations
that the model drops because a value of dependent, exog, weights or other_effects is
missing are not used, so they can be missing as they could before, and so can the
cells of a grid that dependent does not have. Raising for them would make code that
gives the right standard errors fail. The ids of the singleton observations that
PanelOLS drops with singletons=False are checked, since they are complete
observations.

PooledOLS, PanelOLS, RandomEffects, BetweenOLS and FirstDifferenceOLS set up the
clusters at the start of fit, so that the error is raised before the model is
estimated. It was raised after the parameters were estimated, which in PanelOLS is
after the projection on the effects. The other errors in the clusters, such as the
shape of the clusters and clusters that vary within an entity, are raised at the same
place.

The Raises sections of the fit methods and of reformat_clusters list the error, the
fit methods have a versionchanged note for 7.1, and the FAQ of the panel models says
that the cluster of an observation that the model uses cannot be missing.

test_missing_clusters.py fits the five models with the clusters in eight forms
(DataFrame, Series, 1-d array, 2-d and 3-d arrays and a DataFrame of the time periods
by the entities, text and categories), and with a missing id in the second of two
columns. The 45 tests that a missing id raises fail without the change. The 88 others
pass with and without it: clusters without missing values give the same standard
errors as before (those of PooledOLS are also computed from the definition), and the
missing ids of the rows that are dropped are not used.
test_grids_without_labels_are_paired_by_position_in_the_grid reversed a grid in which
the cells without a row are NaN, which moved the NaN to rows that the panel has. The
cells are filled before they are reversed.

#789 made the IV and system models raise a ValueError when the cluster ids have
missing values. The panel models did not. They replace the cluster ids by the codes
of their categories before the covariance is computed, and the code of a missing
value is -1, so the observations with a missing id became a cluster of their own. In
a PooledOLS example with 120 observations and 7 missing ids, the standard error of
the constant changed from 0.0855 to 0.0900 without an error. BetweenOLS and
FirstDifferenceOLS, which need the clusters to be constant within an entity, failed
in other ways: BetweenOLS ignored a missing number if another observation of the
entity had an id, FirstDifferenceOLS ignored a missing id in the first period of an
entity and otherwise raised the error of pandas about converting NaN to an integer,
and text and categories gave "clusters must not vary within an entity" or "clusters
must be identical for values used to compute the first difference".

reformat_clusters, which every panel model uses to place the clusters on the rows of
dependent, now raises the ValueError of the IV and system models, "clusters cannot
contain missing values.", before text and categories are replaced by codes. This
covers every form of clusters: a pandas object or PanelData with the index of
dependent, a 1-d array or list, a grid (a 2-d or 3-d array, a DataFrame of the time
periods by the entities, or a DataArray), and text and categories, with one or two
variables.

Only the observations that the model keeps are checked. The ids of the observations
that the model drops because a value of dependent, exog, weights or other_effects is
missing are not used, so they can be missing as they could before, and so can the
cells of a grid that dependent does not have. Raising for them would make code that
gives the right standard errors fail. The ids of the singleton observations that
PanelOLS drops with singletons=False are checked, since they are complete
observations.

PooledOLS, PanelOLS, RandomEffects, BetweenOLS and FirstDifferenceOLS set up the
clusters at the start of fit, so that the error is raised before the model is
estimated. It was raised after the parameters were estimated, which in PanelOLS is
after the projection on the effects. The other errors in the clusters, such as the
shape of the clusters and clusters that vary within an entity, are raised at the same
place.

The Raises sections of the fit methods and of reformat_clusters list the error, the
fit methods have a versionchanged note for 7.1, and the FAQ of the panel models says
that the cluster of an observation that the model uses cannot be missing.

test_missing_clusters.py fits the five models with the clusters in eight forms
(DataFrame, Series, 1-d array, 2-d and 3-d arrays and a DataFrame of the time periods
by the entities, text and categories), and with a missing id in the second of two
columns. The 45 tests that a missing id raises fail without the change. The 88 others
pass with and without it: clusters without missing values give the same standard
errors as before (those of PooledOLS are also computed from the definition), and the
missing ids of the rows that are dropped are not used.
test_grids_without_labels_are_paired_by_position_in_the_grid reversed a grid in which
the cells without a row are NaN, which moved the NaN to rows that the panel has. The
cells are filled before they are reversed.
The 7.1 notes said that only the IV and system models raise for cluster ids with missing values. The panel models now raise too, and the ids of observations that are dropped for other missing values may still be missing.
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.68%. Comparing base (5ae0cd4) to head (46d8da5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #845   +/-   ##
=======================================
  Coverage   99.68%   99.68%           
=======================================
  Files         198      199    +1     
  Lines       37727    37810   +83     
  Branches     2430     2432    +2     
=======================================
+ Hits        37608    37691   +83     
  Misses         51       51           
  Partials       68       68           
Flag Coverage Δ
adder 99.67% <100.00%> (+<0.01%) ⬆️
subtractor 99.67% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bashtage
bashtage merged commit 94310eb into main Oct 15, 2026
25 checks passed
@bashtage
bashtage deleted the lm71-bug8-panel-nan-clusters branch October 15, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant