GH-48561: [C++][Parquet] Optimize column reader by fusing definition level decoding with counting - #51109
Open
Shockp wants to merge 3 commits into
Open
GH-48561: [C++][Parquet] Optimize column reader by fusing definition level decoding with counting#51109Shockp wants to merge 3 commits into
Shockp wants to merge 3 commits into
Conversation
Add GetBatchAndCount support to RleRunDecoder, BitPackedRunDecoder, RleBitPackedDecoder, and BitPackedDecoder. The new operation decodes values while also counting occurrences of a target value. RLE runs compute the count directly from the run value and length, avoiding an additional scan of the decoded output. Bit-packed runs count the values after unpacking them into the output buffer. Add tests covering decoded output, matching counts, partial batches, and mixed RLE/bit-packed runs.
…nting Use the new decode-and-count operation when reading definition levels. Add LevelDecoder::DecodeAndCount to decode levels while counting occurrences of the maximum definition level, preserving the existing level validation. Update the column reader to use the matching count directly when determining the number of physical values to decode, removing the separate std::count pass over the definition level buffer.
Add benchmarks comparing separate definition level decoding and counting with the fused DecodeAndCount path. Cover RLE and bit-packed encodings across different maximum levels, batch sizes, and level repeat counts.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Definition levels in the Parquet column reader are currently decoded into
a buffer and then scanned separately with
std::countto determine howmany physical values need to be decoded.
This adds an extra pass over the decoded definition levels.
For RLE-encoded levels, the number of matching values can often be
determined directly from the run value and run length while the output is
being materialized.
What changes are included
GetBatchAndCountsupport to the RLE and bit-packed decoders.metadata without scanning the decoded output.
LevelDecoder::DecodeAndCount, preserving the existing levelvalidation.
levels.
std::countpass from the column reader.Benchmarks
parquet-column-reader-benchmark, pinned to one CPU, 20 repetitions.The improvement increases with RLE run length because matching values can
be counted from run metadata instead of rescanning the materialized output.
The bit-packed path remains approximately neutral.
Closes #48561.