[C++][ORC] Writing struct<struct<…>> with a null inner struct shifts the child values #55618

Writing a nested struct to ORC loses data when the inner struct has a null but the outer struct has none. Every child value after the null row is shifted by one, and the first shifted value is written as 0 (or "" for strings).

import pyarrow as pa, pyarrow.orc as orc
inner = pa.struct([("c", pa.int32())])
t = pa.table({"o": pa.array([{"i": {"c": 1}}, {"i": None}, {"i": {"c": 3}}], pa.struct([("i", inner)]))})
orc.write_table(t, "/tmp/a.orc")
orc.read_table("/tmp/a.orc").to_pylist()
# [{'o': {'i': {'c': 1}}}, {'o': {'i': None}}, {'o': {'i': {'c': 0}}}]   expected c = 3

The ORC Java reader (Spark) returns the same wrong values, so the file itself is wrong.

The same happens wherever a struct with nulls sits below another struct that has no nulls in the batch, also through a list or a map:

S = pa.struct([("c", pa.int32())])
pa.array([{"l": [None, {"c": 1}, {"c": 2}]}], pa.struct([("l", pa.list_(S))]))   # reads back [None, {c: 0}, {c: 1}]
pa.array([{"m": [(1, None), (2, {"c": 2})]}], pa.struct([("m", pa.map_(pa.int32(), S))]))   # reads back (2, {c: 0})
pa.array([[{"i": {"c": 1}}, {"i": None}, {"i": {"c": 3}}]], pa.list_(pa.struct([("i", S)])))   # reads back c = 0

Not affected: a top-level list or map<k, struct> with null elements, list<list<…>> and struct<list> with null lists, and any case where the outer struct also has a null in the batch. A round trip over a few thousand random nested schemas outside that shape found no other differences.

Seen on pyarrow 12.0.1 through 26.0.1 (Linux and Windows).

It looks like a struct's null mask is not passed on to its children when the enclosing struct reports no nulls, so the child gets an extra value for each null row. The writer side is WriteStructBatch in cpp/src/arrow/adapters/orc/util.cc.

Found while cross-checking the same files across several readers.