Skip to content

Short-circuit over components without recursion or allocations - #377

Open
aplavin wants to merge 3 commits into
wide_rowsfrom
perf_allcols
Open

aplavin wants to merge 3 commits into
wide_rowsfrom
perf_allcols

Conversation

@aplavin

@aplavin aplavin commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Following #370's map replacement, here we use @nall and @nany from Base to fix more cases of huge overhead on wide StructArrays.

Same direction as #358, but:

  • all affected functions, not just roweq, with @generated code localized in one place (next to existing @ntuple)
  • for both homo- and heterogeneous coltypes
  • also simplify findconsistentvalue and index_type given that the straightforward solution is now also the efficient one

MWE:

using StructArrays
k = rand(1:5, 100_000)
t = StructArray(ntuple(i -> isodd(i) ? k .+ i : Float64.(k), 40))  # 40 key columns, Int and Float64

collect(StructArrays.finduniquesorted(t))      # group row indices by key, as Makie's groupby does
# current  532 ms, 494 MiB
# pr358    509 ms, 494 MiB
# this pr   28 ms,  24 MiB

Saves on compile time as well:

using StructArrays
k = rand(1:5, 1_000)
t = StructArray(ntuple(i -> isodd(i) ? k .+ i : Float64.(k), 256))  # 256 key columns, Int and Float64
p = sortperm(t)

collect(StructArrays.finduniquesorted(t, p))   # first call, includes compilation
# current  14.6 s
# pr358    14.8 s
# this pr  0.54 s

collect(StructArrays.finduniquesorted(t, p))   # second call
# current  0.60 s,  517 MiB
# pr358    0.61 s,  517 MiB
# this pr  0.5 ms,   40 KiB

aplavin and others added 3 commits November 12, 2026 14:13
Add `_allcols` and `_anycols`, unrolled `&&`/`||` chains built with `Base.Cartesian.@nall`/`@nany`. Unlike `all`/`any` and recursion over `tail`, they stay inferrable and allocation-free for any number of components, including NamedTuples, and keep early stopping. Use them in `roweq` (grouping), `issparse` and `broadcast_unalias`.

Co-authored-by: Kristoffer Carlsson <kcarlsson89@gmail.com>
Co-authored-by: Kristoffer Carlsson <kcarlsson89@gmail.com>
Replaces the recursion over components. The special case for homogeneous
component tuples (only check the first component) was added in #194 because
the recursion was not inferred beyond a few elements; without recursion it is
no longer needed, and mixed-type wide tuples now compile faster too.

Co-authored-by: Kristoffer Carlsson <kcarlsson89@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant