Repository navigation
Use symmetric Python linkage with one-to-one assignment - #323
Closed
MaykThewessen wants to merge 7 commits into
Closed
MaykThewessen wants to merge 7 commits into
MaykThewessen wants to merge 7 commits into
Conversation
Implements the architecture proposed in PyPSA#287: deterministic matching on EIC (Energy Identification Code) runs before Duke fuzzy matching in compare_two_datasets(). Plants sharing an EIC code are matched with certainty, then removed from the Duke input so the fuzzy matcher only handles the residual. This provides a robustness guarantee against co-located plant confusion. For example, in the Eemshaven harbour area (Netherlands): OPSD "Eemscentrale Ec" (Natural Gas, 1929 MW, 6 EIC codes) ENTSOE "Eems" (Natural Gas, 1931 MW, same 6 EIC codes) ENTSOE "Eemshaven" (Hard Coal, 1580 MW, different EIC) ENTSOE "Eemshaven" (Natural Gas, 1410 MW, different EIC) EIC matching deterministically pairs the gas plants via their shared codes, preventing any possibility of Duke merging them with the nearby coal plant based on name/geo similarity. Similarly, Borssele nuclear (OPSD: 492 MW, ENTSOE: 485 MW) is locked to its correct cross-source pair via EIC 49W000000000054X, independent of fuzzy name matching against nearby Borssele wind/coal entries. Integration test: 431 deterministic EIC matches between OPSD and ENTSOE (28% of ENTSOE), reducing Duke's workload for those pairs to zero. Closes PyPSA#287 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
for more information, see https://pre-commit.ci
Member
|
These pull requests would have a higher chance of success if they were not stacked and had smaller diffs. The various cross-dependencies of PRs makes it nearly impossible to review appropriately. Also, why open another PR on |
Contributor
Author
Closing in favour of #301. The linkage fixes from this PR are offered there as separate commits: #301 (comment) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace Java DUKE with the Python rapidfuzz engine from #301, preserving the exact-EIC phase from #289. Name scoring uses the lower directional token total so reversing sources does not change scores. Sparse bipartite assignment selects maximum-total-score one-to-one links, with canonical ordering for deterministic ties. Token encoding is vectorized.
Remove bundled Java jars, XML matching files and CI Java setup. Add rapidfuzz and rename parallel_duke_processes to parallel_processes. Matching outputs change.
This draft includes #289 as a prerequisite and offers a reviewable integration of #301 with symmetry and assignment repairs. It does not claim the benchmark accuracy reported for the original #301 scoring rule. The inherited 0.85 threshold still needs independent calibration.
Validation: the extracted EIC, matcher and JRC follow-up pass 64 offline tests together. Four matcher regressions failed against the incoming #301 implementation before these repairs. Local integration inventory validation completed, but source vintages differ from the published snapshot.