Add reverse order support to MultiScan (#14876)

[jaykorean]

Summary:

Add reverse scan support to MultiScanArgs/MultiScan so callers can iterate bounded scan ranges in descending order. Plumb reverse scan state through DBIter, BlockBasedTableIterator, and MultiScanIndexIterator, including reverse prefetch windowing and async I/O support. Update db_bench multiscanrandom to honor --reverse_iterator and report rows scanned so reverse MultiScan and iterator baselines can be compared on row throughput.

Add db_stress --multiscan_reverse coverage so the prepared MultiScan path can be stressed with SeekForPrev()/Prev() against a control iterator.

Also fix the table child iterator path to accept reverse SeekForPrev() targets above a file's prepared ranges. MergingIterator and LevelIterator can legally seek every child to the current range limit; a child file may only have lower prepared ranges, so treating that as invalid caused intermittent SeekForPrev target is outside prepared ranges exceptions in the benchmark.

Differential Revision: D109357730

[meta-codesync]

@jaykorean has exported this pull request. If you are a Meta employee, you can view the originating Diff in D109357730.

[github-actions]

✅ clang-tidy: No findings on changed lines

Completed in 548.0s.

[github-actions]

Codex Code Review - OBSOLETE

Superseded by a newer AI review. Expand to see the original review.

🟡 Codex Code Review

Auto-triggered after CI passed — reviewing commit d253e4d

---

❌ Codex review failed before producing findings.

WARNING: proceeding, even though we could not create PATH aliases: Refusing to create helper binaries under temporary dir "/tmp" (codex_home: AbsolutePathBuf("/tmp/codex-home"))
error: the argument '--base <BRANCH>' cannot be used with '[PROMPT]'

Usage: codex exec review --commit <SHA> --base <BRANCH> --title <TITLE> --model <MODEL> --config <key=value> --dangerously-bypass-approvals-and-sandbox --output-last-message <FILE> [PROMPT]

For more information, try '--help'.

---

ℹ️ About this response

Generated by Codex CLI.
Review methodology: claude_md/code_review.md

Limitations:

• Codex may miss context from files not in the diff
• Large PRs may be truncated
• Always apply human judgment to AI suggestions

Commands:

• /codex-review [context] — Request a code review
• /codex-query <question> — Ask about the PR or codebase

[github-actions]

Claude Code Review - OBSOLETE

Superseded by a newer AI review. Expand to see the original review.

✅ Claude Code Review

Auto-triggered after CI passed — reviewing commit d253e4d

---

Summary

This PR adds reverse scan support to MultiScan, plumbing reverse state through DBIter, BlockBasedTableIterator, and MultiScanIndexIterator. The implementation is well-structured and correctly mirrors the forward path with appropriate reverse semantics. Good test coverage including unit, table-reader, stress, and benchmark integration.

High-severity findings (0):

No high-severity findings.

Full review (click to expand)

Findings

🟡 MEDIUM

M1. Reverse Prev() lacks out-of-bound signaling for multi-file scans -- document design decision -- multi_scan_index_iterator.cc

• Issue: In the forward path, when Next() crosses a scan range boundary, SetExhausted() is called, setting scan_range_exhausted_ = true. FindBlockForward() checks IsScanRangeExhausted() to set is_out_of_bound_ = true, allowing LevelIterator to stop the current file. In the reverse Prev() path, hitting the scan range start simply sets valid_ = false without calling SetExhausted(). This works because for reverse scans, DBIter manages range boundaries externally via iterate_lower_bound_, so the per-file out-of-bound signaling is not needed.
• Suggested fix: Add a comment in MultiScanIndexIterator::Prev() explaining why SetExhausted() is not needed for reverse (DBIter manages range transitions via SeekForPrev with updated bounds), to prevent future confusion.

M2. next_scan_idx_ not updated in reverse path -- potential future hazard -- multi_scan_index_iterator.cc

• Issue: The forward path uses next_scan_idx_ to track expected scan ranges. The reverse path uses cur_scan_idx_ instead. next_scan_idx_ stays at 0. HasMoreScanRanges() checks next_scan_idx_ < block_index_ranges_per_scan_.size(), which would return stale results during reverse iteration. Currently HasMoreScanRanges() is only called in FindBlockForward(), so this is safe.
• Suggested fix: Consider adding an assert or guard in HasMoreScanRanges() that it's only called for forward scans, or document the limitation.

M3. db_stress verify_reverse_multiscan lambda doesn't check iterator status -- db_stress_test_base.cc

• Issue: The verify_reverse_multiscan lambda compares keys between iter and cmp_iter but does not check iter->status().ok() before accessing iter->key(). If iter has an error status, iter->Valid() may be false but the error is not reported. The outer loop checks status after Prev() but the lambda's initial call (after SeekForPrev) runs before that check.
• Suggested fix: Add a status check at the top of the lambda before comparing validity:

  if (!iter->status().ok()) {
    fprintf(stderr, "Reverse MultiScan error: %s\n",
            iter->status().ToString().c_str());
    diverged = true;
    thread->stats.AddErrors(1);
    thread->shared->SetVerificationFailure();
    return;
  }

M4. lower_bound_ update in slow path is fragile -- db/multi_scan.cc

• Issue: In MultiScanIterator::operator++(), the slow path (iterator re-creation due to bound type mismatch) only updates upper_bound_ and iterate_upper_bound. For reverse scans, lower_bound_ and iterate_lower_bound also need updating. Currently safe because validation requires all reverse ranges to have upper bounds (making the slow path unreachable), but future relaxation of validation would break reverse slow-path silently.
• Suggested fix: Add a comment noting the slow path doesn't support reverse, or add reverse bound handling defensively.

🟢 LOW / NIT

L1. Repeated reverse_multi_scan computation -- block_based_table_iterator.cc

• Issue: multi_scan_read_set_ && multi_scan_index_iter_ && multi_scan_index_iter_->scan_opts()->reverse is computed identically in SeekForPrev(), SeekToLast(), and Prev().
• Suggested fix: Extract to bool IsReverseMultiScan() const.

L2. cfh_ zero-initialized but db_ is not -- include/rocksdb/multi_scan.h:257

• Issue: The diff adds {} to cfh_ initialization but db_ remains uninitialized. Consider initializing db_ consistently.

L3. Missing early_exit distinction in op_logs for reverse -- db_stress_test_base.cc

• Issue: The early_exit op_logs entries ("E" and "R") are the same for forward and reverse, making debugging slightly harder.

Cross-Component Analysis

Context	Executes?	Assumptions hold?	Notes
WritePreparedTxnDB	Yes	Yes	read_callback_ handles visibility
User-defined timestamps	Yes	Yes	CompareWithoutTimestamp used throughout
ReadOnly DB	Yes	Yes	Read-only, no write assumptions
Prefix seek	No	N/A	MultiScan requires total_order_seek
Multi-level (LevelIterator)	Yes	Yes	SeekForPrev accepts above-range targets
Concurrent writers	Yes	Yes	Snapshot isolation via ReadOptions
FIFO/Universal compaction	Yes	Yes	No compaction-specific assumptions
BlobDB	Yes	Yes	DBIter handles blob dereferencing

Positive Observations

• Clean architectural layering: DBIter drives range transitions, MultiScanIndexIterator handles per-file block navigation, BlockBasedTableIterator bridges the two.
• The prefetch_start_idx_/prefetch_max_idx_ windowing for reverse prefetch is well-designed, prefetching from the end of blocks.
• IsPrefetchedBlock() helper consolidates range checks and prevents off-by-one errors.
• ReadSet index mapping (block_idx - prefetch_start_idx_) correctly translates indices.
• Comprehensive validation: reverse requires upper bounds, Seek blocked on reverse, SeekForPrev blocked on forward.
• Good defensive fix: accepting SeekForPrev targets above a file's prepared ranges handles MergingIterator/LevelIterator edge cases.
• Strong test coverage: unit test with 3 ranges across multiple files, negative test for missing bounds, table-reader-level test, stress test with control iterator, db_bench integration.

---

ℹ️ About this response

Generated by Claude Code.
Review methodology: claude_md/code_review.md

Limitations:

• Claude may miss context from files not in the diff
• Large PRs may be truncated
• Always apply human judgment to AI suggestions

Commands:

• /claude-review [context] — Request a code review
• /claude-query <question> — Ask about the PR or codebase

[github-actions]

Codex Code Review - OBSOLETE

Superseded by a newer AI review. Expand to see the original review.

🟡 Codex Code Review

Auto-triggered after CI passed — reviewing commit 828c9e8

---

❌ Codex review failed before producing findings.

WARNING: proceeding, even though we could not create PATH aliases: Refusing to create helper binaries under temporary dir "/tmp" (codex_home: AbsolutePathBuf("/tmp/codex-home"))
error: the argument '--base <BRANCH>' cannot be used with '[PROMPT]'

Usage: codex exec review --commit <SHA> --base <BRANCH> --title <TITLE> --model <MODEL> --config <key=value> --dangerously-bypass-approvals-and-sandbox --output-last-message <FILE> [PROMPT]

For more information, try '--help'.

---

ℹ️ About this response

Generated by Codex CLI.
Review methodology: claude_md/code_review.md

Limitations:

• Codex may miss context from files not in the diff
• Large PRs may be truncated
• Always apply human judgment to AI suggestions

Commands:

• /codex-review [context] — Request a code review
• /codex-query <question> — Ask about the PR or codebase

[github-actions]

Claude Code Review - OBSOLETE

Superseded by a newer AI review. Expand to see the original review.

✅ Claude Code Review

Auto-triggered after CI passed — reviewing commit 828c9e8

---

Summary

This PR adds reverse scan support to MultiScan, plumbing the reverse flag through DBIter, BlockBasedTableIterator, and MultiScanIndexIterator. The design is sound overall, with good validation at each layer and proper stress test coverage. One high-severity bug was found: CopyConfigFrom does not propagate the reverse flag to per-file MultiScanArgs, which breaks reverse MultiScan for any scan spanning multiple SST files.

High-severity findings (1):

• [options.h:2084] CopyConfigFrom does not copy the reverse field, causing per-file MultiScanArgs to always have reverse=false. This silently breaks reverse MultiScan for multi-file scans.

Full review (click to expand)

Findings

🔴 HIGH

H1. CopyConfigFrom does not propagate reverse -- include/rocksdb/options.h:2084

• Issue: MultiScanArgs::CopyConfigFrom() copies io_coalesce_threshold, max_prefetch_size, use_async_io, and io_dispatcher, but does not copy the reverse field. This method is called in db/version_set.cc:1357 to propagate configuration to per-file MultiScanArgs during LevelIterator::Prepare().
• Root cause: The reverse field was added to MultiScanArgs but CopyConfigFrom was not updated to include it.
• Impact: When LevelIterator::Prepare() creates per-file scan arguments at version_set.cc:1357, each file's MultiScanArgs will have reverse=false regardless of the original setting. Consequently, BlockBasedTableIterator::Prepare() creates a MultiScanIndexIterator with scan_opts_->reverse == false. All the reverse-aware code paths in MultiScanIndexIterator (SeekForPrev, Prev, SeekToLast) check scan_opts_->reverse and will fail or be unreachable. The BlockBasedTableIterator::SeekForPrev check multi_scan_index_iter_->scan_opts()->reverse will return false, causing ResetMultiScan() to be called, destroying the prepared state.
• Affected paths: Both sync (version_set.cc:1943) and async (version_set.cc:1376) prepare paths.
• Suggested fix: Add reverse = other.reverse; to CopyConfigFrom():

  void CopyConfigFrom(const MultiScanArgs& other) {
    io_coalesce_threshold = other.io_coalesce_threshold;
    max_prefetch_size = other.max_prefetch_size;
    use_async_io = other.use_async_io;
    reverse = other.reverse;  // ADD THIS
    io_dispatcher = other.io_dispatcher;
  }

🟡 MEDIUM

M1. Scan::Reset signature breaks API consistency -- include/rocksdb/multi_scan.h:75

• Issue: Scan::Reset(Iterator*, bool) removes the old single-argument signature without a default parameter, unlike Scan(Iterator*, bool reverse = false) which preserves backward compatibility via default. This is inconsistent. While Reset is only called internally (in MultiScanIterator::operator++), the class is in a public header.
• Suggested fix: Add a default parameter: void Reset(Iterator* db_iter, bool reverse = false).

M2. Reverse stress test requires two flags but no documentation -- db_stress_tool/db_stress_test_base.cc:2054

• Issue: The reverse MultiScan path in db_stress requires BOTH --multiscan_reverse AND --test_backward_scan to be set. This compound condition (FLAGS_multiscan_reverse && FLAGS_test_backward_scan) is easy to miss in CI configuration. If only --multiscan_reverse is set, the reverse path is silently skipped.
• Suggested fix: Either (1) make --multiscan_reverse sufficient by itself, or (2) add a validation warning when --multiscan_reverse is set without --test_backward_scan.

M3. No test coverage for reverse MultiScan with max_prefetch_size limit -- test files

• Issue: The ReverseMultiScanPrepare test sets max_prefetch_size = 10 * 1024 * 1024 (10MB), effectively unlimited. No test exercises the reverse prefetch window boundary where prefetch_start_idx > 0 and PrefetchLimitReached is returned. The forward path has MultiScanPrefetchSizeLimit for this; a reverse equivalent is missing.
• Suggested fix: Add a test with a small max_prefetch_size that forces the reverse prefetch window to exclude some blocks, verifying PrefetchLimitReached behavior.

M4. No test for cross-direction error paths -- test files

• Issue: No test verifies that calling Seek() on a reverse MultiScan or SeekForPrev() on a forward MultiScan returns InvalidArgument. The validation code is present in DBIter but untested.
• Suggested fix: Add test cases for both cross-direction error conditions.

🟢 LOW / NIT

L1. Repeated reverse_multi_scan computation -- block_based_table_iterator.cc:207,282,329

• Issue: The same 3-pointer-dereference check is computed at the top of SeekForPrev(), SeekToLast(), and Prev():

  const bool reverse_multi_scan = multi_scan_read_set_ &&
                                  multi_scan_index_iter_ &&
                                  multi_scan_index_iter_->scan_opts()->reverse;

  Consider caching this as a member bool set during Prepare() and cleared in ResetMultiScan().

L2. cfh_ initialization -- include/rocksdb/multi_scan.h:257

• Issue: ColumnFamilyHandle* cfh_{}; uses brace initialization (new), while the old code left it uninitialized. The change is correct and an improvement, but it's a drive-by fix unrelated to reverse scan support.

L3. Wasteful double-release in destructor -- multi_scan_index_iterator.cc:35

• Issue: The destructor releases all blocks in [prefetch_start_idx_, prefetch_max_idx_), but blocks already released during Prev() iteration will be released again. ReadSet::ReleaseBlock is idempotent (second call is a no-op), so this is safe but wasteful. The forward path has the same characteristic.
• Suggested fix: Track which blocks have been released, or adjust the destructor range based on cur_idx_. Low priority since idempotent.

Cross-Component Analysis

Context	Does code execute?	Assumptions hold?	Action needed?
Multi-file scans (LevelIterator)	YES	NO -- CopyConfigFrom missing reverse	Fix H1
WritePreparedTxnDB	YES	YES -- visibility handled by read_callback	None
ReadOnly DB	YES	YES -- no writes needed	None
User-defined timestamps	YES	YES -- CompareWithoutTimestamp used consistently	None
Prefix seek	Possible	Untested -- SeekForPrev + prefix extractor	Worth testing
FIFO/Universal compaction	YES	YES -- scan is read-only	None
Async IO reverse	YES	Partially -- no async reverse test	Add test

Positive Observations

• Clean separation of forward/reverse logic at each layer
• Good use of valid_ flag instead of sentinel index values for termination
• Proper validation in DBIter::SeekForPrev that blocks cross-direction calls
• ValidateScanOptions correctly requires upper bounds for reverse
• Stress test includes divergence detection with detailed error output
• db_bench reports rows_scanned for meaningful performance comparison
• The fix for SeekForPrev targets above a file's prepared ranges (gap handling) is correctly implemented in MultiScanIndexIterator::SeekForPrev
• Block double-release safety verified: ReadSet::ReleaseBlock, CachableEntry::Reset, and ReleaseAsyncIOForBlock are all idempotent
• Copy/move constructors and assignment operators all properly copy the reverse field

---

ℹ️ About this response

Generated by Claude Code.
Review methodology: claude_md/code_review.md

Limitations:

• Claude may miss context from files not in the diff
• Large PRs may be truncated
• Always apply human judgment to AI suggestions

Commands:

• /claude-review [context] — Request a code review
• /claude-query <question> — Ask about the PR or codebase

[github-actions]

🟡 Codex Code Review

Auto-triggered after CI passed — reviewing commit 41a36f0

---

❌ Codex review failed before producing findings.

WARNING: proceeding, even though we could not create PATH aliases: Refusing to create helper binaries under temporary dir "/tmp" (codex_home: AbsolutePathBuf("/tmp/codex-home"))
error: the argument '--base <BRANCH>' cannot be used with '[PROMPT]'

Usage: codex exec review --commit <SHA> --base <BRANCH> --title <TITLE> --model <MODEL> --config <key=value> --dangerously-bypass-approvals-and-sandbox --output-last-message <FILE> [PROMPT]

For more information, try '--help'.

---

ℹ️ About this response

Generated by Codex CLI.
Review methodology: claude_md/code_review.md

Limitations:

• Codex may miss context from files not in the diff
• Large PRs may be truncated
• Always apply human judgment to AI suggestions

Commands:

• /codex-review [context] — Request a code review
• /codex-query <question> — Ask about the PR or codebase

[github-actions]

✅ Claude Code Review

Auto-triggered after CI passed — reviewing commit 41a36f0

---

Summary

Solid implementation of reverse MultiScan plumbing through the iterator stack. The core design (bool reverse, SeekForPrev+Prev, ascending range specification) is consistent with RocksDB patterns. One high-severity bug found: CopyConfigFrom() does not propagate the reverse field, which will break reverse MultiScan for multi-file scans via LevelIterator. One medium-severity API breaking change in Scan::Reset().

High-severity findings (1):

• [options.h:2084] CopyConfigFrom() does not copy the reverse field, causing per-file MultiScanArgs to default to reverse=false when propagated through LevelIterator.

Full review (click to expand)

Findings

🔴 HIGH

H1. CopyConfigFrom() missing reverse field — include/rocksdb/options.h:2084

• Issue: MultiScanArgs::CopyConfigFrom() copies io_coalesce_threshold, max_prefetch_size, use_async_io, and io_dispatcher but does NOT copy the new reverse field. This method is called from db/version_set.cc:1357 to propagate scan configuration to per-file MultiScanArgs objects used by LevelIterator::Prepare().
• Root cause: The reverse field was added to copy/move constructors and assignment operators (lines 1997, 2003, 2013, 2025, 2089) but was missed in CopyConfigFrom().
• Impact: When a reverse MultiScan spans multiple SST files, each file's BlockBasedTableIterator::Prepare() will receive MultiScanArgs with reverse=false. This means:
  • MultiScanIndexIterator won't support SeekForPrev/Prev (they check scan_opts_->reverse)
  • Prefetch window will be calculated from the beginning (forward) instead of the end (reverse)
  • The SeekForPrev call in BlockBasedTableIterator::SeekForPrev() will use index_iter_->Seek() instead of index_iter_->SeekForPrev(), leading to incorrect block positioning
  • The Prev() path will trigger ResetMultiScan(), falling back to non-prepared iteration
• Suggested fix: Add reverse = other.reverse; to CopyConfigFrom():

  void CopyConfigFrom(const MultiScanArgs& other) {
    io_coalesce_threshold = other.io_coalesce_threshold;
    max_prefetch_size = other.max_prefetch_size;
    use_async_io = other.use_async_io;
    reverse = other.reverse;  // ADD THIS
    io_dispatcher = other.io_dispatcher;
  }

🟡 MEDIUM

M1. Scan::Reset() API breaking change — include/rocksdb/multi_scan.h:75

• Issue: The Reset() method signature changed from Reset(Iterator*) to Reset(Iterator*, bool reverse) without a default parameter. While Scan is marked EXPERIMENTAL and the constructor has a default (bool reverse = false), the Reset method does not, creating an inconsistency. Any external code calling Reset() with one argument will fail to compile.
• Root cause: Default parameter was added to constructor but not to Reset().
• Suggested fix: Add a default: void Reset(Iterator* db_iter, bool reverse = false).

M2. Destructor may double-release blocks — multi_scan_index_iterator.cc:35

• Issue: The destructor now releases blocks in [prefetch_start_idx_, prefetch_max_idx_) unconditionally. However, MultiScanIndexIterator::Prev() calls ReleaseBlock(cur_idx_) before decrementing, and SeekToBlockIdxReverse() calls ReleaseBlocks(block_idx + 1, cur_idx_ + 1). The destructor does not track which blocks have already been released — it iterates the entire prefetch range.
• Root cause: ReleaseBlock() calls read_set_->ReleaseBlock() which likely tolerates double-release, but this should be verified.
• Suggested fix: Verify that ReadSet::ReleaseBlock() is idempotent. If not, track released blocks or only release unreleased blocks in the destructor.

M3. Stale lower bound during iterator recreation — db/multi_scan.cc:~170

• Issue: When MultiScanIterator::operator++ transitions between ranges in reverse mode, the iterator recreation (when upper bound presence changes) happens before SeekCurrentRange() updates the lower bound. The new DBIter may be created with a stale iterate_lower_bound.
• Root cause: The order of operations: check if iterator recreation is needed, recreate if so, then call SeekCurrentRange() which updates both bounds.
• Suggested fix: Update lower bound before iterator recreation for reverse scans.

🟢 LOW / NIT

L1. reverse_multi_scan computed redundantly — block_based_table_iterator.cc:208,283,331

• Issue: The same 3-condition check is computed separately in SeekForPrev(), SeekToLast(), and Prev().
• Suggested fix: Cache the reverse flag in BlockBasedTableIterator during Prepare().

L2. Missing test for multi-file reverse scan

• Issue: No test explicitly verifies reverse scan across multiple SST files at the BlockBasedTableIterator level. Given H1, such a test would have caught the CopyConfigFrom bug.

L3. scan_range_exhausted_ asymmetry undocumented

• Issue: Forward scan uses SetExhausted()/IsScanRangeExhausted() for range boundary signaling. Reverse bypasses this entirely, relying on DBIter's lower bound. This is correct but the design choice should be documented.

Cross-Component Analysis

Context	Does code execute?	Assumptions hold?	Action needed?
WritePreparedTxnDB	YES	YES	Safe
ReadOnly DB	YES	YES	Safe
User-defined timestamps	YES	YES	Safe
Trie index	YES	YES — kInbound correct for Prev	Safe
LevelIterator multi-file	YES	NO — CopyConfigFrom bug (H1)	Must fix
FIFO/Universal compaction	YES	YES	Safe

Positive Observations

• Clean separation of forward/reverse logic with clear scan_opts_->reverse gates
• Thorough validation in DBIter matching the forward path's pattern
• Well-designed reverse prefetch window calculation
• Proper stress test verification with control iterator
• Sound reasoning for TrieIndex PrevAndGetResult kInbound change

---

ℹ️ About this response

Generated by Claude Code.
Review methodology: claude_md/code_review.md

Limitations:

• Claude may miss context from files not in the diff
• Large PRs may be truncated
• Always apply human judgment to AI suggestions

Commands:

• /claude-review [context] — Request a code review
• /claude-query <question> — Ask about the PR or codebase