feat: add gold layer, a normalized trip-centric warehouse for Nov 2023 - #33
Merged
Merged
Conversation
* feat: substitute nearest export for missing GTFS tables * perf: partition silver tables by load period * fix: match GTFS referential tests to donor export * fix: set a fallback raw_data_root so tests can collect in CI
* fix: skip empty raw AVL day files * fix: detect colliding AVL month folder names * fix: read AVL rows as strings and drop corrupted ones
…#26) Adds five more bronze reference sources alongside vehicle_dictionary, all raw files already sitting in DICIONARIO_VEICULOS/ but never ingested by anything: device_dictionary (a new device_id-to-vehicle_number export, a bridge distinct from vehicle_id/id_veiculo), two older vehicleid/numbus and id/carro dictionaries, and two more veiculos_atuais.csv-shaped snapshots (2018 and an older "antigo" export). Kept as separate bronze sources rather than merged into vehicle_dictionary since none of their id spaces are confirmed compatible with the current live file's. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Adds silver loaders for the five new vehicle dictionary bronze sources (device_dictionary, vehicle_dictionary_legacy, vehicle_dictionary_legacy2, vehicle_dictionary_2018, vehicle_dictionary_antigo), alongside the existing vehicle_dictionary loader, all living in the single silver/vehicle_dictionary.py module. Each gets its own partitioned table with a unique index on whichever column was verified unique within a snapshot, kept separate rather than merged since none of their id spaces are confirmed compatible with the live file's. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…28) Renames all six vehicle dictionary silver tables (vehicle_dictionary, device_dictionary, vehicle_dictionary_legacy, vehicle_dictionary_legacy2, vehicle_dictionary_2018, vehicle_dictionary_antigo) to a shared dictionary_* prefix (dictionary_vehicle, dictionary_device, dictionary_legacy, dictionary_legacy2, dictionary_2018, dictionary_antigo), matching the avl_*/afc_*/gtfs_* naming so they sort together alphabetically. Bronze source names and CLI source arguments are unchanged, only the physical silver table names. load-silver-reference now echoes the actual table it wrote to instead of assuming it matches the source argument. This breaks gold's existing int_vehicle_dictionary source reference (silver.vehicle_dictionary no longer exists), left as-is per explicit direction since gold isn't being worked on right now. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
) veiculos_atuais.csv was previously documented and named as the one "live" file in the dictionary family, re-read on every future ingest as it got updated in place. All five raw dictionary files (including this one) are about to be deleted from raw_data_root now that they're fully captured in bronze, so that framing is no longer accurate for any of them -- rewords the adapters/contracts/silver docstrings accordingly and renames ingest -> ingest_vehicle and load -> load_vehicle for symmetry with ingest_legacy/load_legacy and the rest, which never had a bare unqualified name to begin with. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Drops the gold/ dbt project, its dbt-core/dbt-postgres dependencies, and every doc/comment reference to the gold layer. No gold schema or objects existed in the local database, so there was nothing to drop there.
The Postgres connection is shared by every schema (silver, validation, ml, ...), not just silver, so the SILVER_ prefix on DB_DSN/DB_USER/ DB_PASSWORD/DB_NAME was stale and misleading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Needed to author and run the Trip Validity model's build notebooks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Notebooks print output intentionally in every cell that reports a result - that's how you see anything in a notebook - so T201 doesn't apply the way it does to library/CLI code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously a ruff check in isolation was easy to mistake for "done," which let ~289 lint findings accumulate across a batch of new notebooks before anyone ran the full hook suite. Documents two gotchas discovered doing that cleanup: prek --all-files only checks git-tracked files, and the ruff hooks lint .ipynb too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four notebooks build the ml schema end to end for November 2023: trips/fares straight from silver.afc_boardings, GTFS route/shape/ schedule matching (the only stage that touches silver.gtfs_*), the per-trip feature table (duration vs. peer/scheduled expectations, path-shape comparison, directional progress correlation), and the final 76-column ML-ready dataset with every relative/normalized column. Every table is self-contained within ml after the GTFS materialization stage - nothing downstream ever rejoins silver. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ts, and local Claude state .pytest_cache was only hidden because pytest drops its own .gitignore inside that directory, which only takes effect after the first test run. Also adding .ipynb_checkpoints/ (new now that the repo has real notebooks) and .coverage/htmlcov/ proactively, plus .claude/ (found untracked and empty, but that's where Claude Code keeps local session state). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The opa superuser was retired (NOLOGIN) in favor of a per-deployment admin account, so the hardcoded opa:opa connection example was wrong.
Both services used to publish on 0.0.0.0, reachable from any device on the same network as the Docker host, not just the tailnet. They now bind to a configurable BIND_HOST, defaulting to 127.0.0.1 for a fresh local setup with no remote access.
Only the bronze-ingest code path actually reads from raw_data_root, so a missing/wrong path now surfaces as a FileNotFoundError there instead of blocking every other use of settings (e.g. db_dsn-only scripts) up front.
vehicle_ts_idx and device_ts_idx now carry latitude/longitude/speed/ odometer as INCLUDE columns, so a per-vehicle time-window lookup (e.g. matching AVL pings to a trip) can be satisfied as an index-only scan instead of fetching the heap page for every matching row.
Resolves each trip's bus_id to an AVL vehicle_id/device_id via silver.dictionary_device (preferred) or silver.dictionary_vehicle (fallback), then materializes every matched trip's GPS trace (ml.trip_validity_trip_positions) within its own time window for fast per-trip lookup later (map rendering).
Renumbered 04_final_dataset.ipynb -> 05_final_dataset.ipynb (now runs after AVL position matching). ml.trip_validity_dataset gains trip_opening_timestamp, trip_closing_timestamp, avl_matched, and avl_match_source, so a trip's own window and AVL-match status don't require a separate lookup.
Both joins are PK lookups (feed_version_date+shape_id, trip_id), so this stays fast without touching silver or scanning anything.
Streamlit app under ml/trip_validity_model/app/ implementing the full active learning loop: calibration/test/seed/active labeling phases, LightGBM + Optuna hyperparameter/feature-selection retraining on a 15/50-label cadence, Platt calibration, and a two-map (GTFS direction I/V) Folium view with a time-graded AVL trail for visual verification. Model runs and labels are persisted to new ml.trip_validity_labels and ml.trip_validity_model_runs tables, with artifacts joblib-dumped to ml/trip_validity_model/artifacts/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…el budget Active-phase draws now split 1/3 uncertain / 1/6 random (diversity) / 1/2 random (calibration or test, whichever is smaller) into train, calibration, and test respectively - training and evaluation grow at the same overall rate instead of evaluation being frozen at 50/50 forever. Each set stops accepting draws once it hits its cap (TRAIN_CAP=250, CALIBRATION_CAP=125, TEST_CAP=125); once full, a draw that would've gone there is redirected to whichever open set is furthest below its own target, so the budget finishes at exactly those numbers rather than merely converging toward them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both derived from the existing trip_date via GENERATED ALWAYS AS ... STORED (ISO 8601 weekday, so 1=Monday...7=Sunday, avoiding Postgres's DOW convention where Sunday=0), matching the pattern silver already uses for PostGIS geometry columns - no separate backfill needed, and they can't drift out of sync with trip_date. Appended (not rewritten) to 05_final_dataset.ipynb so a full rebuild still reproduces them, and made the ALTER idempotent since the live table was migrated directly rather than by re-running the notebook. Added to the app's CATEGORICAL_FEATURES so the next milestone retrain can pick them up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t-reload Skipped trips are now tracked in-memory for the current app run (st.session_state.skipped_trip_ids) and excluded from both the random and uncertainty-ranked draw paths, so a skip never resurfaces within the same session - but nothing is written to the database, so restarting the app clears it, matching "pretend I never saw it." Also fixes a real crash hit during this session: registry.save_run was pickling PlattCalibrator/ModelConfig instances directly, which breaks with a PicklingError if those modules get hot-reloaded (e.g. by Streamlit's file watcher during dev) between when an instance is created and when it's pickled - the reloaded module's class is a distinct object with the same name. Now stores plain data (the underlying sklearn model, feature list, hyperparameter dict) and reconstructs fresh instances on load, with backward-compatible loading for artifacts already written in the old pickled-instance format. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Once both eval sets are capped, every draw feeds train regardless of the random/uncertain coin flip, so there's no more eval pool to protect from uncertainty bias. Uncertain probability rises from 1/3 to 3/4 at that point, verified via simulation to land on the intended ~75/25 split without disturbing the exact 250/125/125 final budget.
Once calibration and test are both capped, every draw feeds train anyway, so there's no more reason to hold back full hyperparameter/ feature optimization for eval-set-growth's sake. landmark_crossed now upgrades every 15-label "cycle" retrain to a full "milestone" one once eval_sets_closed(counts) is true, instead of only every 50th label.
One row per trip (940,988): trip_id, bus_id, route_id, date, start/end timestamps, the GTFS feed + shape_id_i/v for a direct join to route_shapes/route_stops, and a single validity flag - sourced from a human label where one exists (500 rows), otherwise from the notebook-06 final model, thresholded at its own saved decision threshold and tagged with a stable model_name/confidence rather than a raw file path. Adds PlattCalibrator.from_params to calibration.py (symmetric with the existing to_params), so this notebook can reconstruct the saved calibrator without manually poking sklearn internals. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One row per AFC fare tap (15,381,356), matched to its trip in ml.trip_validity_final regardless of that trip's validity: event_id, card_id, integration_type, passenger_type, fare_paid, subsidy, and the tap timestamp (fare_tapped_at). Built directly from silver.afc_boardings joined to ml.trip_validity_trips by the same natural key notebook 01 already used, not by rejoining through event_id - '0' is a sentinel for "no id assigned" in the source data and is not unique (223,399 of 15,381,356 rows), so it can't serve as a key. fare_event_id is a fresh GENERATED ALWAYS AS IDENTITY surrogate key instead, rebuildable exactly like trip_id/fare_id already are; the raw event_id (duplicates included) is kept as a plain, non-unique column for traceability. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
validate/test ran plain `uv sync`, pruning the ml group entirely, while pyproject.toml's ty config includes ml/trip_validity_model/app in the type-check surface - every one of its numpy/pandas/sklearn/folium imports was failing to resolve in CI. Never caught before now since this is the first time CI actually ran against this branch's app code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feat: train the Trip Validity model
First two notebooks for the Bus Matching model: ml.bus_matching_candidate_pairs builds every plausible bus_id to device_id pairing from the two vehicle dictionaries (corroborated against ml.trip_validity_final and silver.avl_pings for November 2023 +/- one day), for a later model to pick real matches out of. ml.bus_matching_avl_positions is a view over the same window, renaming and reformatting a few AVL columns (route_code to route_id, direction to heading_degrees) to match this project's existing ml table conventions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
10 of the 140,655,482 rows in the window have a raw direction of 360 or 9360 (both exact multiples of 360), which isn't a valid compass heading. The view now maps direction >= 360 to 0 instead of passing it through. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-date Tier 1 feature parquet files are a rebuildable cache, not source of truth -- keep them out of git the same way the model artifacts and DB tables already carry the real state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Precompute pipeline for the bus matching model: bus-date contestedness (93% of bus-dates share a route/time window with another bus, so most of the month needs real disambiguation, not just a lone AVL match), the spatiotemporal grid+time-bucket signature tables that turn candidate blocking into a cheap integer hash join, and the resulting ml.bus_matching_candidates table (dictionary-sourced + top-N blocking matches per bus-date). Also points ty at ml/bus_matching_model/app the same way it already does for ml/trip_validity_model/app, so these notebooks' imports from gtfs_cache resolve. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trip-level labeling UI (ml/bus_matching_model/app/) driving a probabilistic belief engine over candidate devices per bus-date, not a hard vote-count rule: each trip decision is noisy evidence in log-odds space, seeded by a Tier 1 heuristic prior until a trained LightGBM model takes over. A bus-date resolves once its top candidate clears a confidence bar and is clearly separated from the runner-up, so one bad click can never resolve anything alone. Key pieces: - belief.py: log-odds evidence combination, cross-bus-date suppression (a device confidently claimed on one bus discounts its plausibility elsewhere that day), temporal-support propagation across dates via a two-pass acyclic design, and an ECE-based trust ramp so the model's prior only earns weight once its own measured calibration is good, not just because time/labels have passed. - db.py: belief/candidate queries, auto-selection (fetch_next_trip) with two modes -- "hardest" (uncertainty sampling, tiebroken toward dictionary-backed candidates) and "random" (sweeps up the ordinary case active learning otherwise starves, added after live coverage numbers visibly jittered between retrains) -- and stopping_signal(), an auto-detected convergence check instead of a fixed label budget. - features.py / gtfs_cache.py: vectorized Tier 1 feature computation (chainage/offset projection, direction correlation, stop coincidence, etc.) against an in-memory GTFS shape cache. - streamlit_app.py: the labeling UI itself -- sticky sidebar decisions, two maps per candidate (one per GTFS direction), score + dictionary- origin shown directly on each candidate on request (an explicit, acknowledged override of the plan's anti-bias hiding). - training.py / calibration.py / registry.py: LightGBM training loop. Platt calibration is disabled on request -- confirmed live that refitting it on a small per-retrain split swung live coverage ~9x with no change in the underlying model, even after an earlier fix for the same instability at smaller scale. Raw model probability feeds belief directly now; test_ece already gates trust on it. Also: pyproject.toml's ty environment.extra-paths now includes this app dir (needed so the precompute notebooks can resolve gtfs_cache), ordered after trip_validity_model/app since the two trees share several module basenames (features.py, db.py, ...) and ty resolves bare sibling imports against whichever path is listed first -- and .pre-commit-config.yaml excludes this app dir from the ty hook entirely, since checking it under that same shared config would break trip_validity_model's own sibling imports instead. Linting/formatting for this app was deferred during active iteration per an explicit request earlier in its build; brought to a clean prek pass now as part of committing it for real. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every retrain's LightGBM artifact from the active-learning loop so far, on request -- same precedent as trip_validity_model's own artifacts/ directory (full per-run history, not just the latest). ml.bus_matching_model_runs already has the corresponding metrics/ metadata row for each of these. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces belief.py's cross-bus-date suppression -- a single, non-iterated pairwise-discount heuristic that can still leave two buses claiming the same device -- with a real joint solve per date: minimum-weight full bipartite matching between buses and devices, so a device is matched to at most one bus that day, enforced by construction rather than approximated. - belief.py: factor the prior+temporal+vote evidence accumulation out of compute_date_beliefs into compute_raw_beliefs, so the global solve can consume the same pre-suppression posterior the heuristic itself approximates, instead of layering a second constraint on top of an already-suppressed one. compute_date_beliefs's own public behavior is unchanged (verified against live data before/after). - assignment.py: builds one date's sparse bus-x-(device+dummy) cost matrix (cost = -log(posterior)) and solves it with scipy.sparse.csgraph.min_weight_full_bipartite_matching -- sized for real per-date graphs (confirmed live: ~1,600 buses x ~1,400 devices, ~16,500 edges, ~0.7% dense), not a dense linear_sum_assignment. Every bus gets its own "assign to none" dummy column (cost = that bus's own NONE_OPTION posterior) so a bus with no good candidate isn't forced onto a bad one. Per-bus margin (plan Section 9.3: re-solve with that bus's edge forbidden, take the cost increase) computed exactly, not approximated -- confirmed live at ~2.3ms/bus, cheap enough to not need a shortcut. - notebooks/06_global_assignment.ipynb: batch job over all 30 dates (per the plan and the labeling app's own docstring, this stays a periodic background job, not part of the live UI), writing ml.bus_matching_global_assignment. Verified live: 41,332 rows (every valid bus-date, including the 1,325 with zero candidates at all, marked method='no_candidates' rather than fed to the solver), zero devices double-assigned on the same date across all 30 dates, and 100% agreement with belief.py's own already-confident picks on the 25,927 bus-dates where it had one -- the new signal is entirely in the ~14,000 bus-dates the heuristic couldn't previously call. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Collapses each device's daily sequence of Section 9 (ml.bus_matching_global_assignment) assignments into (bus_id, device_id, from_date, to_date) intervals. Scope confirmed live before choosing an approach: of 1,437 devices with at least one assigned day, 1,407 (97.9%) stick to exactly one bus all month -- nothing to smooth. Only 30 devices (60 device-bus pairs) ever show more than one bus, and 25 of those 60 pairs are a single day. At this scale a transparent, inspectable rule-based smoother (app/smoothing.py) is a better fit than the plan's suggested HMM/changepoint-detection machinery, not an under-implementation of it -- confirmed correct against every one of the hand-inspected real sequences before trusting it. Two rules, both gated by a cross-device conflict check: gap bridging (same bus on both sides of unassigned days -> bridge, unless the plan's own "missing data is never negative evidence" doesn't apply because Section 9 directly confirmed a *different* device on that bus during the gap) and isolated one-day-deviation correction (a single-day blip surrounded by the same bus on both sides gets folded in and logged). Caught and fixed a real bug before committing: an early version without the conflict check bridged a device with only 2 solved days 14 days apart straight through 6+ days where the same bus was solidly confirmed to a different device -- a genuine double-claim, not a smoothing nicety. Added a same-bus/different-device overlap check to the notebook specifically because it caught this; both that check and the pre-existing same-device overlap check now come back 0 across the full month (1,492 intervals, 30,121 total assigned days accounted for exactly, 23 isolated-deviation corrections logged to ml.bus_matching_smoothing_corrections for later inspection). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…me-bus-stickiness Three changes from a domain expert's direct feedback on why coverage wasn't reaching every bus, and how confident we should be in the signals we already have: 1. No-AVL company exclusion (notebooks/06): silver.dictionary_device companies where 100% of known devices never ping in November 2023 -- confirmed live to be exactly COOTRAPS and Fretcar, computed fresh each run rather than hardcoded -- are excluded from the solver entirely and written method='no_avl_data'. These buses have real AFC trips but their assigned devices structurally cannot appear in any AVL-based match (verified across 7 months of pings, and independently corroborated by a completely separate prior matching attempt in scratch.* that also found nothing for them). Previously these ~249 buses looked like ordinary, wrong-looking "none" results indistinguishable from a real modeling gap. 2. Dictionary corroboration as real evidence (belief.py, db.py): a dictionary-sourced pair is meaningfully more trustworthy than a blocking-only candidate -- not certain, and weaker specifically when a bus has multiple disagreeing dictionary-sourced devices -- confirmed on request from a domain expert, not a guess. Was previously a UI-only badge with zero weight in the actual belief math or Section 9's cost matrix; now a full log-odds term (DICTIONARY_PRIOR_WEIGHT/DICTIONARY_BOOST_ODDS_*), stronger when both dictionary tables independently agree on the same pair. db.fetch_all_candidates_with_scores now carries n_dictionary_sources and bus_has_dictionary_conflict through from ml.bus_matching_candidate_pairs' origin column. 3. Same-bus stickiness extension (smoothing.py, notebooks/07): a device essentially never changes bus mid-month, so a bus with exactly one distinct confirmed device across all its Section 10 intervals gets that device extended across its *entire* running range, gated by the same cross-device conflict check as gap-bridging. Two real bugs caught and fixed before trusting this: (a) checking only the extending bus's own running days for conflicts missed one that fell entirely on days that bus doesn't run at all, silently double-claiming a device -- fixed by scanning the full calendar range between candidate days, not just running-day positions; (b) the conflict lookup was built from Section 9's raw per-day output, so it still saw the pre-correction value on days Section 10 had already resolved as noise -- fixed with a new smoothing.build_device_owner that applies corrections first. Both confirmed by re-running the same overlap-freeness checks Section 9 established. Net effect, verified live: solved bus-dates with a real device jumped from 30,121/40,007 (75%) to 31,981/34,581 (92%) after excluding no-AVL buses from the denominator; buses with at least one matched day now 1,430 of 1,490 AVL-eligible buses (96%), up from 1,428 of 1,730 total before this work made the eligible population itself honest. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dictionary corroboration boost (added earlier this session) was a flat additive bonus applied regardless of how well the candidate's own GPS/model score looked -- confirmed live this let a dictionary-backed candidate win over a clearly better-scoring competitor in 118 of ~32,000 solved bus-dates, one case as stark as a 0.91-scoring candidate losing to a 0.15-scoring dictionary pick purely from the boost. On request, from the same domain expert: "if a dictionary says a given bus but it's clearly off in the metrics, it shouldn't win -- only if the dictionary one is the highest fit or close to it." The boost in belief.py now only applies when a candidate's own score is within DICTIONARY_RELATIVE_MARGIN (0.2) of the best score among that bus-date's candidates; otherwise it contributes nothing, so a dictionary pick can no longer resurrect a candidate the evidence has already rejected. Verified live: violations dropped from 118 to 18, and the residual 18 checked out as legitimate joint-assignment effects, not remaining bugs -- in every case checked, the higher-scoring competitor had been allocated to a *different* bus that same day by Section 9's joint solve, so the dictionary-backed pick was genuinely the best option still available to this bus, not a case of the boost overriding better evidence. Re-ran notebooks 06 and 07 with the fix; coverage numbers essentially unchanged (this only affects which device wins in a small number of close calls), overlap-freeness re-verified at 0 both directions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Five more retrains' worth of artifacts from continued active-learning labeling, same precedent as the earlier bulk commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ures Eight new per-(bus, device, date) features, taking the set from 23 to 31, plus a committed parallel build script. All are inputs to a trained model rather than hand-weighted terms -- part of moving every trust decision from a constant I picked to a coefficient learned from labels. New signals: - median_heading_consistency: the AVL feed's own compass heading vs the bearing of the GTFS segment each point matched. gtfs_cache now precomputes seg_bearing_deg and exposes the matched segment index via project_points_full. Independent of the chainage-vs-time correlation, so it stays meaningful on trips too short or sparse for a stable one. - median_direction_agreement: AFC's own binary route_direction (from ml.trip_validity_trips, previously unused by this model) against the candidate's best-fit GTFS direction. On request, the I/V-to-0/1 mapping is deliberately ARBITRARY rather than researched -- if it is backwards the model simply learns a negative coefficient, which is equally informative. That only works because it feeds a trained model. - median_fare_stationary_fraction / median_fare_near_stop_m: the plan's Section 3.3 discriminator for two buses on the same route minutes apart -- fares are collected while stopped, so a true pairing has its fare taps land where the device was stationary near a route stop. - frac_device_points_in_windows, frac_device_moving_points_in_windows, first_activity_gap_seconds, last_activity_gap_seconds: day-level continuity, computed against the bus's *full* trip list rather than the 8-trip sample (using the sample would undercount by half). Verified on 2023-11-01 before committing to a full rebuild: separation between likely-correct and likely-wrong pairs is 1.92 std for heading consistency and 1.75 std for fare-near-stop (89m median vs 1,883m) -- both strong. direction_agreement separates weakly on its own (0.13 std), which is expected: its value is in interaction (disambiguating two devices running the same route in opposite directions), not marginal separation, and the model is free to weight it accordingly. Frechet/Hausdorff is deliberately still not implemented and the module docstring now says so explicitly: it is a per-trip DP that across the month's ~3.3M trip-candidate pairs is the one Tier 2 item that would push a rebuild into many hours, and direction_correlation already covers the order-sensitivity it was specified for. scripts/build_features.py parallelizes by date (dates are fully independent) and excludes the two confirmed no-AVL companies up front. Measured: 320s for one date serially, so ~20 min for the month at 8 workers. Writes to artifacts/features_v2/ rather than overwriting features/, so current production results stay reproducible alongside. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The layer that retires belief.py's hand-set constants. Every signal that used to arrive as a guessed weight -- dictionary trust, temporal continuity, cross-bus suppression -- is now a plain feature the model weights from labels instead. - pair_features.py: 73 month-level features per (bus, device), built by aggregating the 31 day features (median/mean/std, so consistency counts as well as level), the day model's own score across the month, dictionary corroboration and both directions of dictionary conflict, and competition on both sides (rank/gap within the bus, rank/gap within the device, mutual-best). Deliberately excludes Section 9/10 output: the pair model is meant to replace the hand-weighted layer that produced those, so feeding them back in would make the final confidence partly a function of the heuristics it exists to retire. - schema.py: ml.bus_matching_pair_labels (three-way correct/wrong/ unsure, keyed by pair for the whole month) and ml.bus_matching_pair_model_runs, kept separate from the day model's run history so the two never mix. - pair_model.py: training (split by bus, never by row, so a bus's positive and its implied negatives can't straddle the split and leak), ranking, the labeling queue, progress counters, and precision_at_threshold -- which is what makes the ship threshold a measured choice rather than another guessed constant. A "correct" verdict implies a negative for every other candidate of that bus, so one click is worth many training rows: verified live at 60 labels -> 4,126 rows (60 pos / 4,066 neg). Caught a real trap before it cost a labeling session: ranking the queue by raw top-to-runner-up margin put eleven zero-evidence buses first, because near-zero minus near-zero is a tiny margin. Their best of ~73 candidates scored 5.7e-7 -- absence of evidence, not ambiguity, and the same trap already fixed at the day level. select_hard_buses now requires MIN_EVIDENCE_SCORE and no_evidence_buses reports the rest as their own countable bucket. Nine of the eleven are 67-prefix buses absent from the dictionary snapshot, so the company-based no-AVL exclusion could not catch them; they need candidate generation or a data source to change, not a human decision. Verified end to end on real data: 120,718 pairs over 1,442 buses, and 1,255 buses (87%) already clear 0.9 on the day-score fallback before any pair label exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
exclusions.py is now the single source of truth for which buses no AVL-based matcher can resolve, imported everywhere so the feature build, pair layer, and notebooks can't disagree. Two independent rules, because neither alone suffices: any dictionary company whose devices never ping (computed live, not hardcoded), and -- on request -- every 67-prefix bus outright. The company rule is dictionary-driven and so blind to vehicles missing from the snapshot: it caught 240 of 249 COOTRAPS buses, and the 9 it missed had gone on to dominate the labeling queue as apparent hardest cases. Fretcar qualifies under the company rule but turns out to have zero valid trips in the period, so it removes nothing. Net: 249 excluded, 1,481 buses in scope. pair_labeler.py is the validation UI (port 8502, alongside the trip labeler on 8501). Layout as requested: candidates across as columns, sampled trips down as rows, one map per cell, so confusing candidates are compared side by side on the same trips rather than judged from memory. Trips are sampled across *different dates* on purpose -- two trips from one morning show the same corridor twice, while spread trips test whether a candidate tracks the bus consistently, which is the actual claim being judged. Shows what was asked for: per-candidate confidence, and progress split into confirmed-by-hand vs model-confident vs remaining, so labeling effort is visibly moving a number. The ship threshold is a slider wired to pair_model.precision_at_threshold, which reports measured precision at whatever cut is chosen -- the threshold is meant to be picked from that number rather than guessed. Verified via AppTest against real data: renders 1,433 buses in scope (1,481 minus the 48 with no candidates at all, which by definition have no pair rows), 1,255 already model-confident, and the first queued bus is a genuinely ambiguous one (top candidates at 0.110 / 0.058 / 0.001). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The day model had a stopping signal; the pair layer had none, so there was no answer to "how do I know when to stop labeling". Adds pair_model.stopping_signal and surfaces it in the UI. Deliberately descriptive rather than prescriptive: it reports the two numbers that actually decide the question -- how many buses are still unsettled, and what precision has been *measured* at the current threshold -- and only calls "done" for the unambiguous case. No invented convergence heuristic; target_precision is a choice made with the measured number in front of you. Refuses to quote a precision figure below MIN_MEASURED_FOR_PRECISION labeled pairs, since precision over a handful of labels would swing wildly with the next click and reads as more certain than it is. Also distinguishes the case that matters most for the remaining tail: when the ambiguous queue empties but buses are still below threshold, those need candidate generation or new data, not more labeling. Checked before writing this: all 1,255 currently-confident buses also have a top-to-runner-up margin >= 0.5, so "confident" and "unambiguous" coincide in practice today. The counter still keys on score alone, and that is deliberate -- ambiguity is already a model input via the competition features, so a separate margin gate would double-count and reintroduce a hardcoded constant. Worth re-verifying once real pair labels exist, since today's scoring is still the day-model fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
After the first 21 real pair labels the UI reported 100% precision,
which is true and meaningless: every label was verdict='correct' AND
was_top_candidate=True, so "when the top pick clears the threshold, is
it right?" was being measured on a sample where only top picks were
ever shown and only confirmations were ever given. It was 21/21 by
construction, and the model still had zero examples of a wrong top
pick.
The missing piece is the plan's own Section 7 item -- "random confident
pairs, small but non-negotiable ... the only honest check on whether
confident predictions are actually right". The ambiguity-ranked queue
structurally cannot surface a *silent* error, because it only ever
shows buses the model is unsure about.
- schema: pair labels gain label_source ('queue' | 'audit'), defaulting
existing rows to 'queue'.
- pair_model.select_audit_buses: uniform random sample of buses already
above the threshold. Random on purpose -- sampling the weakest of the
confident ones would be biased the other way and overstate errors.
- pair_model.audit_precision: precision over audit rows only. Pooling
the two sampling regimes yields a number that means neither thing.
- stopping_signal now keys off audit precision, so "can I stop?" can
never be answered by confirmations alone.
- UI: a sampling-mode radio (teach vs measure), and the two precision
figures reported separately -- the audit one labeled as the number to
trust, the pooled one explicitly framed as a floor.
Verified against the real 21 labels: the app now reports "0 audit
labels, precision not measurable" instead of a misleading 100%.
Also worth flagging from this fit: n_dictionary_sources came out #2 of
73 by importance. Plausibly an artifact of 21 positives (dictionary
backing is rare among the 1,709 implied negatives, so it separates
easily) but worth re-checking at 60+ labels. The four dictionary
*conflict* features sit at importance 0, so nothing is leaning on them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three gaps found after the first 20 audit labels came back 20/20. 1. The audit was uniform over confident buses, which sounds unbiased but isn't useful: confirmed live that 84.6% of buses above 0.90 sit at >=0.999, so a uniform draw spends nearly every label on near-certain buses. All 20 audit labels landed in the 0.999-1.01 band, so the reported 100% said nothing about the 216 buses in 0.90-0.99 where errors would actually live -- which is exactly why every audited bus looked obviously correct. select_audit_buses now samples evenly across AUDIT_BANDS (uniform *within* each band, so each band's precision stays unbiased for that band), and audit_precision reports per-band alongside the overall figure. 2. Metrics came from a single 70/30 split. With a few dozen labeled buses that lands on a handful and swings with which ones. cross_validated_metrics adds grouped 5-fold CV (grouped by bus, same leakage reason as the split), reporting mean +/- std so "AUC 1.000" can be told apart from "AUC 0.87 +/- 0.19". Current fit: 1.000 +/- 0.000, i.e. genuinely separable rather than lucky -- though on labels that are still all confirmations. 3. The pair model existed only in Streamlit session state, so it vanished on restart and no downstream step could reproduce a score. save_pair_run/load_latest_pair_run persist it to artifacts/pair_models/ plus ml.bus_matching_pair_model_runs (CV metrics ride in hyperparameters, keeping the run table the same shape as the day model's). Also: train_pair_model now refits on all rows before returning, since a model trained on 70% of an already-small label set is strictly worse at the job it actually does; the holdout and CV numbers still describe held-out performance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…etection Labeling converged: 1,424 of 1,481 in-scope buses resolved at >=0.90 with unanimous margin (every runner-up scored at least 0.9 below the winner), and the remaining tail turned out to have real, explainable structure rather than being genuinely stuck. This closes it out into `ml.bus_matching_final_pairs`, following the same never-silently-drop convention `ml.bus_matching_global_assignment` already established (explicit `method` per row), just at the month grain: every one of the 1,730 buses that ran gets exactly one row, or one row per interval for a detected device swap. The labeling session surfaced a case the one-device-per-bus assumption can't represent on its own: a device genuinely gets swapped for another mid-month, and averaging both devices' day-scores across the whole month makes two clearly-correct-on-their-half devices look like a 50/50 tie. Four such buses were caught by hand and marked verdict='unsure' rather than forced into a wrong single answer. final_output.detect_split tells a real swap apart from a genuine "I don't know" purely from the day-score time series -- one device confidently winning its days, a single clean changeover (not several, not interleaved), a different device confidently winning the rest. Verified against all four hand-caught swaps (12225, 12502, 35252, 35403): every one auto-detects with the exact date boundary already confirmed by hand, including 12502's genuine overlap day (both devices confident on Nov 14, resolved to whichever scored higher). Not every 'unsure' is a swap -- some are the user genuinely not knowing, and forcing those through the same detector would fabricate a boundary that isn't there. Two buses came back "no confident days for any candidate" and are bucketed as needs_review rather than guessed at; one turned out to have only ever had one real confident device and is resolved_after_review instead of split. Full breakdown from this run (1,730 buses, 1,734 rows): 1,345 pair_model, 249 excluded_no_avl, 77 hand_confirmed, 48 no_candidates, 8 split_detected (4 buses), 4 no_evidence, 2 needs_review, 1 resolved_after_review, 0 below_threshold -- nothing left stuck between "confident" and "explicitly flagged." Also persists an official pair-model run via pair_model.save_pair_run (previously only fit ad hoc in Streamlit session state) -- run #11 on this data: 84 pair labels, 5-fold CV AUC 1.000 +/- 0.000. Committing the pair-model checkpoints accumulated during labeling alongside it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bus_matching_final_pairs is bus-centric -- for every bus, which device. That leaves the reverse question unanswered: a device can be genuinely active all month and still never be claimed anywhere. Confirmed live: of 1,493 devices that pinged at all in November 2023, 62 are claimed by no bus -- not a trivial edge case, several have 30 days of activity and 80,000+ pings. device_coverage.py builds ml.bus_matching_unclaimed_devices, one row per unclaimed device with an explicit reason so none are left silently unexplained: 44 never_blocked (blocking never considered them for any bus), 18 lost_competition (they competed for a real bus and another device won), 0 blocked_only_to_excluded_bus this run. Surfaced something concrete for the residual pass rather than more unexplained absence: 5 of the 62 are heavily active (11k-114k pings, 3-30 days) *and* have a dictionary entry pointing at a specific bus, yet scored near zero (~0.00003-0.00007) as a candidate for that exact bus. E.g. device ep1-428109738 has 113,914 pings across all 30 days and the dictionary says it belongs to bus 30162 -- which is independently sitting in the no_evidence bucket on the bus-side table -- but the day model rejected the pairing outright. Either the dictionary is wrong here or there's a real data-quality issue with this device; worth a specific look rather than lumping it in with the buses that have no evidence at all. Wired into notebook 08 alongside the bus-side table, executed live: 1,730 buses accounted for, 62 unclaimed devices found and persisted, five flagged as worth a specific look. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The device-coverage investigation traced why 5 heavily-active, dictionary-backed devices scored near zero as candidates for their own bus. Two turned out to be genuinely weaker/bad matches (correctly rejected). Three were not: ep1-428109738/30162, ep1-428111212/30713, and ep1-428103763/30915 are each their bus's #1 candidate AND that device's #1 candidate (is_mutual_best), day-score ~0.9999 on every day with data, beating the next-best rival by 3-4 orders of magnitude -- yet scored ~0.00003-0.00007 by the pair model. Root cause: the pair model's positive training labels have frac_bus_days_with_data mean=0.984, median=1.0 (almost every confirmed match had near-complete AVL coverage). These three have real coverage gaps (55-73%) from ordinary intermittent pinging, which the model had essentially never seen as a positive and extrapolated catastrophically for. 3 of the 4 `no_evidence` buses were this exact bug, not real absence of evidence -- confirmed by checking the 4th (36975) shows a genuinely different, actually-empty profile (best candidate's day-score never exceeds 0.0007, no mutual-best, no dictionary). Labeled the three directly at the user's request rather than through the UI, since the diagnosis already established they're correct with more certainty than a blind queue click would. label_source = 'investigation', kept distinct from 'queue'/'audit' so it can never pollute audit_precision's unbiased sample -- these were selected *because* they were already suspected correct, the opposite of a random draw. Verified: scores flip from ~0.00005 to ~0.9997+ on retrain (run #15, holdout AUC 0.9999, 5-fold CV AUC 1.0000). Effect on the final tables, confirmed live: no_evidence 4 -> 1, hand_confirmed 77 -> 80, unclaimed devices 62 -> 59. Notebook 08 re-executed end to end with all sanity checks (bus-count match, no overlapping intervals, all 4 known swaps still detected) passing against the updated data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bus 21517 was flagged as one of the two needs_review buses -- an 'unsure' label the user gave during audit that overrode a confidently resolved pair_model score. Investigating it directly (raw day features, 1 real day of data, dictionary-backed) showed it genuinely is the correct match, not the same coverage-bias bug fixed earlier for the other 3 (those had many partial-coverage days; this has exactly one real day, thin but genuine). Relabeled correct per the user's instruction, deliberately deferred rebuilding until now. Rebuild confirmed the fix landed cleanly: hand_confirmed 80 -> 81, needs_review 2 -> 1 (only 20290 left, a genuine single-trip sentinel bus), unclaimed devices 59 -> 58 (ep1-428103634 now claimed). All sanity checks still pass: bus-count match, zero overlapping intervals, all 4 known device swaps still detected at their known boundaries. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feat: bus-to-device matching pipeline (Sections 6-13) with learned pair model
Builds a proper star schema (buses, companies, routes/directions/feeds, shapes, stops, journeys, trips, cards, passenger types, fares, and a unified trip_positions fact table) entirely from silver plus two ML outputs: ml.trip_validity_final (which trips are valid, and their matched GTFS route/shape) and ml.bus_matching_final_pairs (which AVL device a bus actually carried, per date range). No other ML table feeds gold, and nothing here re-derives trip validity or bus/device identity. Real Postgres PK/FK/CHECK constraints throughout, not just documentation or dbt tests. Bus is the only vehicle identity that's ever a key; device is reachable only via bus_device_intervals as informational history. Location lives solely in trip_positions (both AVL- and fare-origin rows, with an origin discriminator), never on fares itself. A few real, empirically-verified corrections baked in: - Route 614's AFC-direction/GTFS-Ida-Volta mapping is reversed relative to every other route (verified against GPS ground truth in a prior session); hardcoded since no other artifact records this fact anymore. - route_direction_stops' canonical stop list per shape is chosen by correlation with the shape's own geometry, not raw trip-count majority -- checked against real November 2023 data first, since majority-vote alone would pick a data-association error over the genuinely correct pattern in at least one real case. - Company names/modalities hand-seeded from the only source that has them (inline notebook comments in ml/trip_validity_model), since no queryable registry exists. Scoped to one month (Nov 2023) as a one-shot build, not yet a general per-period reloadable pipeline like bronze/silver's replace_period -- deliberately deferred. Split into two independently-committed phases (build_base, build_trip_positions) rather than one all-or-nothing transaction, specifically so the expensive trip_positions join could be diagnosed with a real EXPLAIN against persisted, ANALYZEd tables and re-run repeatedly while tuning it. That tuning surfaced two real Postgres gotchas worth remembering: a join against a table with no literal WHERE bound in the query text scanned the source table's entire multi-year history instead of the one relevant month (43GB+ of spilled temp files), and a correlated per-row lookup needs an explicit OFFSET 0 inside a LATERAL subquery to stop the planner from flattening it back into a full-table sort/hash join that ignores each row's own narrow filter until far too late. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Built end to end against real November 2023 data:
Test plan
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com