Repository navigation
TST: Compare the summaries of effects-only results as words and numbers - #846
Merged
Merged
Conversation
The R-squared of a PanelOLS model that only has effects is 0 in exact arithmetic. It is computed as 1 - residual_ss / total_ss, which is 0, 1.11e-16 or -2.22e-16 depending on the rounding of the platform, and the summary prints the difference: "-2.22e-16" in one result and "0.0000" in the other. The two fits that test_effects_only.py compares (a formula and the same model without one) differ in the last bit on Intel macOS, and comparing their summaries as text failed about 25 tests in every Intel macOS job of the wheels build, in test_formula_equals_the_model_without_regressors, test_formula_equals_the_model_with_fit_options and test_formula_with_other_effects. assert_same_results compared the summaries and the string of each test statistic as text. It now compares the words of the two texts, with the numbers replaced by a placeholder, and the numbers as numbers, with rtol=1e-8 and atol=1e-12. The absolute tolerance is the one that the same function gives to the R-squared and the other attributes of the results. The confidence intervals use the check of the other tables, labels and assert_allclose, instead of DataFrame.equals.
test_summary_with_a_zero_standard_error fits a system whose first equation is exact, so that the standard error is 0, the t-statistic is infinite and the summary has to format "inf". With a normal regressor the fit is only exact if the cross products x'x and x'y round alike. They do with the linear algebra of Linux, Windows and Apple silicon. On Intel macOS the standard error is 7.5e-17, the t-statistic is 2.7e16 and no inf is formatted, so that the test failed in every Intel macOS job of the wheels build. Comparing the standard error with a tolerance is not enough, because the test also needs the infinite t-statistic. The regressor is now 1 or -1 in 64 observations. Its sum of squares and its product with the dependent variable of the exact equation, 64 and 128, are integers, so that the parameter is exactly 2 and the standard error exactly 0, whatever the order of the sums of the platform. The standard error is compared with assert_allclose and the t-statistic with np.isinf, as before.
Four tests compare two fits of the same data that are equal in exact arithmetic. On Intel macOS the two fits differ in the last bits and the comparisons were too strict. The tolerances come from the differences that the fits show when the data change in the last bit, and are far below the differences of a wrong result. test_label_resolution.py::test_asset_pricing_inputs failed for LinearFactorModelGMM in the three pairs of resolutions. The model minimizes its objective with BFGS and a finite difference gradient, so that fits that only differ by rounding stop at estimates that differ by up to 1e-7, not by 1e-16. The parameters differed by 3.7e-8, against rtol=1e-10. This model now uses rtol=1e-6, and the other models keep 1e-10. test_case_insensitive_options.py::test_mixed_case_names_give_the_results_of_lower_case_names failed once across the Intel macOS jobs, for the same reason. The relative differences of the smallest elements of its covariance reach 2e-7, which is above the default rtol of 1e-7 in half of the fits whose data change in the last bit. This model now uses rtol=1e-5, and the other models keep the default. test_gmm_kernel_bandwidth.py compared two ways to set the same bandwidth with assert_array_equal and ==. The J statistic of the debiased weights was 1.564448925745796 in one and 1.5644489257457959 in the other. assert_identical is now assert_same_results and uses rtol=1e-8 for the parameters, the standard errors, the covariance, the weight matrix and the J statistic. Changing the data in the last bit changes them by up to 4e-10, in the covariance of the Parzen kernel. test_grid_inputs.py::test_weights_in_a_grid_are_those_of_the_rows_in_every_model compared the residuals with rtol=1e-12 and no absolute tolerance. The relative error of a residual of 1e-4 reaches 2e-11 when the data change in the last bit, and this failed for RandomEffects in 15 of 384 such fits. compare_results now uses rtol=1e-10 for the parameters, the covariance and the residuals, and atol=1e-12 for the residuals.
The wheels of linearmodels 7.1 are built for Intel macOS (macos-15-intel) as well. In the trial build every Intel macOS job failed 29 to 33 tests, while the Linux, Windows and Apple silicon jobs passed, because no job of this repository runs on Intel macOS and so nothing here noticed that the linear algebra of that platform rounds differently. This adds one matrix entry to cron-test.yml, macos-15-intel with Python 3.14 (cp314). The six jobs of the existing matrix are unchanged.
The tests that failed on Intel macOS compared two results that are equal in exact arithmetic exactly, and the last bits of a result depend on the linear algebra of the platform. This applies the same rule to the other tests: a float that is computed is compared with assert_allclose and a tolerance that follows from the difference that is expected, and the numbers in a text are compared as numbers. The comparisons were found by searching the tests for ==, assert_array_equal, assert_equal and array_equal on floats, and for assert_allclose with rtol=0 or atol=0, and by running the whole suite with the last bit of the floats of these comparisons changed. By package: - asset_pricing: the J statistic of a just identified model, the moment covariance of an exactly identified one, the standard errors that are set to 0 for variances that are 0 up to rounding, the t-statistics and p-values that are recomputed, and the parameters after the missing values are dropped. - iv: the kernel weights, s2 of the covariances, the covariances of the aliases of a kernel, the R-squared of a formula and of the model it stands for, the refits of IVJIVE, and the tests of the default bandwidth of IVGMM and IVGMMCUE, like the one of IVSystemGMM that was changed before. - panel: the zeros of a model that only has effects, and the variance of the effects with and without the small sample adjustment. - shared and system: the statistic of a quadratic form that is 0, the p-values of statistics that are infinite, r_squared, the weight matrix of the kernel, and the transform of a linear constraint. - tests: the R-squared that is recomputed from the sums of squares, and the models that only have a constant. The tolerance is 1e-10 for results of closed form from the same data (1e-12 for scalars and elementwise formulas), atol=1e-12 for what is 0 in exact arithmetic, and relative to the largest element for matrices (assert_close_matrix), since the elements of a matrix that are close to 0 have larger relative errors. IVGMMCUE minimizes its objective with BFGS. Changing the data in the last bit changes its estimates by up to 6e-6, the covariance by 3e-6, the R-squared by 6e-9 and the J statistic by 1e-13. The tests use rtol=1e-4 for the estimates and the covariance, 1e-6 for the R-squared and 1e-8 for the J statistic. A bandwidth that differs by 1 changes the estimates by 4e-2 or more. assert_same_text (tests/_utility.py) compares two texts as their words, with the numbers replaced by a placeholder, and as their numbers, with rtol=1e-8 and atol=1e-12. It replaces the comparison of whole texts of the summary of a pickled result, the representation of the results of the Hausman test, the clones of the test statistics and str(res) of a model that only has effects. These stay exact, because exact is the property that is tested: values that are copied, converted or reshaped without arithmetic (the containers of the data, the options of a model, pickled statistics), indicators of 0 and 1 and counts that are stored as floats, the equal weights that normalize to exactly 1, which the models test for, the covariance that is symmetric to the last bit, a series of integers that sums to 0, the exact zeros of sparse matrices in assert_allclose, and the tests of the format of the text of a number that is a literal.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #846 +/- ##
=======================================
Coverage 99.68% 99.68%
=======================================
Files 198 198
Lines 37727 37768 +41
Branches 2430 2431 +1
=======================================
+ Hits 37608 37649 +41
Misses 51 51
Partials 68 68
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The R-squared of a PanelOLS model that only has effects is 0 in exact
arithmetic. It is computed as 1 - residual_ss / total_ss, which is 0, 1.11e-16
or -2.22e-16 depending on the rounding of the platform, and the summary prints
the difference: "-2.22e-16" in one result and "0.0000" in the other. The two
fits that test_effects_only.py compares (a formula and the same model without
one) differ in the last bit on Intel macOS, and comparing their summaries as
text failed about 25 tests in every Intel macOS job of the wheels build, in
test_formula_equals_the_model_without_regressors,
test_formula_equals_the_model_with_fit_options and
test_formula_with_other_effects.
assert_same_results compared the summaries and the string of each test statistic
as text. It now compares the words of the two texts, with the numbers replaced by
a placeholder, and the numbers as numbers, with rtol=1e-8 and atol=1e-12. The
absolute tolerance is the one that the same function gives to the R-squared and
the other attributes of the results. The confidence intervals use the check of
the other tables, labels and assert_allclose, instead of DataFrame.equals.