Skip to content

GH-48561: [C++][Parquet] Optimize column reader by fusing definition level decoding with counting - #51109

Open
Shockp wants to merge 3 commits into
apache:mainfrom
Shockp:gh-48561-fuse-definition-level-count
Open

GH-48561: [C++][Parquet] Optimize column reader by fusing definition level decoding with counting#51109
Shockp wants to merge 3 commits into
apache:mainfrom
Shockp:gh-48561-fuse-definition-level-count

Conversation

@Shockp

@Shockp Shockp commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Definition levels in the Parquet column reader are currently decoded into
a buffer and then scanned separately with std::count to determine how
many physical values need to be decoded.

This adds an extra pass over the decoded definition levels.

For RLE-encoded levels, the number of matching values can often be
determined directly from the run value and run length while the output is
being materialized.

What changes are included

  • Add GetBatchAndCount support to the RLE and bit-packed decoders.
  • For repeated RLE runs, compute the matching count directly from run
    metadata without scanning the decoded output.
  • For bit-packed runs, count the values immediately after unpacking them.
  • Add LevelDecoder::DecodeAndCount, preserving the existing level
    validation.
  • Use the fused decode-and-count operation when reading Parquet definition
    levels.
  • Remove the separate std::count pass from the column reader.
  • Add targeted benchmarks comparing the separate and fused approaches.

Benchmarks

parquet-column-reader-benchmark, pinned to one CPU, 20 repetitions.

Encoding Max level Batch Repeat Separate Fused Change
RLE 1 1024 1 5555.5 ns 5383.5 ns 3.10% faster
RLE 1 1024 7 13464.7 ns 10972.1 ns 18.51% faster
RLE 1 1024 1024 4111.3 ns 971.7 ns 76.36% faster
RLE 3 1024 1 5500.5 ns 5003.0 ns 9.04% faster
RLE 3 2048 1 5384.0 ns 4778.2 ns 11.25% faster
RLE 3 1024 7 14184.0 ns 11833.4 ns 16.57% faster
BitPack 1 1024 1 4001.3 ns 3953.2 ns 1.20% faster
BitPack 1 1024 7 3996.6 ns 3949.6 ns 1.18% faster
BitPack 1 1024 1024 3974.4 ns 3988.0 ns 0.34% slower
BitPack 3 1024 1 3940.1 ns 3974.0 ns 0.86% slower
BitPack 3 2048 1 3895.6 ns 3881.6 ns 0.36% faster
BitPack 3 1024 7 3956.2 ns 3919.4 ns 0.93% faster

The improvement increases with RLE run length because matching values can
be counted from run metadata instead of rescanning the materialized output.
The bit-packed path remains approximately neutral.

Closes #48561.

Shockp added 3 commits August 31, 2026 17:52
Add GetBatchAndCount support to RleRunDecoder, BitPackedRunDecoder, RleBitPackedDecoder, and BitPackedDecoder.

The new operation decodes values while also counting occurrences of a target value. RLE runs compute the count directly from the run value and length, avoiding an additional scan of the decoded output. Bit-packed runs count the values after unpacking them into the output buffer.

Add tests covering decoded output, matching counts, partial batches, and mixed RLE/bit-packed runs.
…nting

Use the new decode-and-count operation when reading definition levels.

Add LevelDecoder::DecodeAndCount to decode levels while counting occurrences of the maximum definition level, preserving the existing level validation.

Update the column reader to use the matching count directly when determining the number of physical values to decode, removing the separate std::count pass over the definition level buffer.
Add benchmarks comparing separate definition level decoding and counting with the fused DecodeAndCount path.

Cover RLE and bit-packed encodings across different maximum levels, batch sizes, and level repeat counts.
@Shockp
Shockp requested review from pitrou and wgtmac as code owners August 31, 2026 19:53
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #48561 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Optimize column reader by fusing definition level decoding with counting

1 participant