Repository navigation
BUG: Raise for nan in cluster ids - #845
Merged
Merged
Conversation
#789 made the IV and system models raise a ValueError when the cluster ids have missing values. The panel models did not. They replace the cluster ids by the codes of their categories before the covariance is computed, and the code of a missing value is -1, so the observations with a missing id became a cluster of their own. In a PooledOLS example with 120 observations and 7 missing ids, the standard error of the constant changed from 0.0855 to 0.0900 without an error. BetweenOLS and FirstDifferenceOLS, which need the clusters to be constant within an entity, failed in other ways: BetweenOLS ignored a missing number if another observation of the entity had an id, FirstDifferenceOLS ignored a missing id in the first period of an entity and otherwise raised the error of pandas about converting NaN to an integer, and text and categories gave "clusters must not vary within an entity" or "clusters must be identical for values used to compute the first difference". reformat_clusters, which every panel model uses to place the clusters on the rows of dependent, now raises the ValueError of the IV and system models, "clusters cannot contain missing values.", before text and categories are replaced by codes. This covers every form of clusters: a pandas object or PanelData with the index of dependent, a 1-d array or list, a grid (a 2-d or 3-d array, a DataFrame of the time periods by the entities, or a DataArray), and text and categories, with one or two variables. Only the observations that the model keeps are checked. The ids of the observations that the model drops because a value of dependent, exog, weights or other_effects is missing are not used, so they can be missing as they could before, and so can the cells of a grid that dependent does not have. Raising for them would make code that gives the right standard errors fail. The ids of the singleton observations that PanelOLS drops with singletons=False are checked, since they are complete observations. PooledOLS, PanelOLS, RandomEffects, BetweenOLS and FirstDifferenceOLS set up the clusters at the start of fit, so that the error is raised before the model is estimated. It was raised after the parameters were estimated, which in PanelOLS is after the projection on the effects. The other errors in the clusters, such as the shape of the clusters and clusters that vary within an entity, are raised at the same place. The Raises sections of the fit methods and of reformat_clusters list the error, the fit methods have a versionchanged note for 7.1, and the FAQ of the panel models says that the cluster of an observation that the model uses cannot be missing. test_missing_clusters.py fits the five models with the clusters in eight forms (DataFrame, Series, 1-d array, 2-d and 3-d arrays and a DataFrame of the time periods by the entities, text and categories), and with a missing id in the second of two columns. The 45 tests that a missing id raises fail without the change. The 88 others pass with and without it: clusters without missing values give the same standard errors as before (those of PooledOLS are also computed from the definition), and the missing ids of the rows that are dropped are not used. test_grids_without_labels_are_paired_by_position_in_the_grid reversed a grid in which the cells without a row are NaN, which moved the NaN to rows that the panel has. The cells are filled before they are reversed.
The 7.1 notes said that only the IV and system models raise for cluster ids with missing values. The panel models now raise too, and the ids of observations that are dropped for other missing values may still be missing.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #845 +/- ##
=======================================
Coverage 99.68% 99.68%
=======================================
Files 198 199 +1
Lines 37727 37810 +83
Branches 2430 2432 +2
=======================================
+ Hits 37608 37691 +83
Misses 51 51
Partials 68 68
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#789 made the IV and system models raise a ValueError when the cluster ids have
missing values. The panel models did not. They replace the cluster ids by the codes
of their categories before the covariance is computed, and the code of a missing
value is -1, so the observations with a missing id became a cluster of their own. In
a PooledOLS example with 120 observations and 7 missing ids, the standard error of
the constant changed from 0.0855 to 0.0900 without an error. BetweenOLS and
FirstDifferenceOLS, which need the clusters to be constant within an entity, failed
in other ways: BetweenOLS ignored a missing number if another observation of the
entity had an id, FirstDifferenceOLS ignored a missing id in the first period of an
entity and otherwise raised the error of pandas about converting NaN to an integer,
and text and categories gave "clusters must not vary within an entity" or "clusters
must be identical for values used to compute the first difference".
reformat_clusters, which every panel model uses to place the clusters on the rows of
dependent, now raises the ValueError of the IV and system models, "clusters cannot
contain missing values.", before text and categories are replaced by codes. This
covers every form of clusters: a pandas object or PanelData with the index of
dependent, a 1-d array or list, a grid (a 2-d or 3-d array, a DataFrame of the time
periods by the entities, or a DataArray), and text and categories, with one or two
variables.
Only the observations that the model keeps are checked. The ids of the observations
that the model drops because a value of dependent, exog, weights or other_effects is
missing are not used, so they can be missing as they could before, and so can the
cells of a grid that dependent does not have. Raising for them would make code that
gives the right standard errors fail. The ids of the singleton observations that
PanelOLS drops with singletons=False are checked, since they are complete
observations.
PooledOLS, PanelOLS, RandomEffects, BetweenOLS and FirstDifferenceOLS set up the
clusters at the start of fit, so that the error is raised before the model is
estimated. It was raised after the parameters were estimated, which in PanelOLS is
after the projection on the effects. The other errors in the clusters, such as the
shape of the clusters and clusters that vary within an entity, are raised at the same
place.
The Raises sections of the fit methods and of reformat_clusters list the error, the
fit methods have a versionchanged note for 7.1, and the FAQ of the panel models says
that the cluster of an observation that the model uses cannot be missing.
test_missing_clusters.py fits the five models with the clusters in eight forms
(DataFrame, Series, 1-d array, 2-d and 3-d arrays and a DataFrame of the time periods
by the entities, text and categories), and with a missing id in the second of two
columns. The 45 tests that a missing id raises fail without the change. The 88 others
pass with and without it: clusters without missing values give the same standard
errors as before (those of PooledOLS are also computed from the definition), and the
missing ids of the rows that are dropped are not used.
test_grids_without_labels_are_paired_by_position_in_the_grid reversed a grid in which
the cells without a row are NaN, which moved the NaN to rows that the panel has. The
cells are filled before they are reversed.