Optimize replication latency by fix the misjudgment of the WAL iterator status(#13260)

[Paragrf]

Description has been recorded in #13260

There are two scenarios where the iterator incorrectly determines that the file has been fully traversed before reaching the actual end, resulting in a "TryAgain" return. However, when traversal is interrupted within a WAL file, subsequent attempts to call SeekToStartSequence can incur significant delays. Our tracking indicates that in such cases, SeekToStartSequence can take between 80 to 200 milliseconds, and RestrictedRead may be executed up to 100,000 times.

case1:

check current_last_seq_ == versions_->LastSequence() twice, but external writes between the two checks may cause the LastSequence to increase, leading to the success of the first check and the failure of the second

Figure 1: double check in nextImpl

Figure 2: first check in RestrictedRead

After addressing this issue, the delay in replication has been significantly optimized, though occasional delay spikes still occur.

Figure 3: replication Pmax(Red line: control group, Orange line: experimental group)

case2:

current_log_reader_->ReadRecord(record, &scratch_) may return false in kEof branch. In certain scenarios, reaching EOF does not necessarily indicate that the file has truly reached its end. We observed this behavior in some custom log info, which also explains the spikes seen in the experimental group in Figure 3.

Although we have not yet pinpointed the specific scenarios that lead to this false EOF, we can prevent this misjudgment by verifying whether a new live WAL file has actually been generated. This issue can be completely solved after adding this check.

Figure 4: replication Pmax(Red line: control group, Orange line: experimental group)