kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6530) Use actual first offset of messages when rolling log segment for magic v2
Date Wed, 07 Mar 2018 23:56:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390468#comment-16390468

ASF GitHub Bot commented on KAFKA-6530:

dhruvilshah3 opened a new pull request #4660: KAFKA-6530: Use actual first offset of message
set when rolling log segment
URL: https://github.com/apache/kafka/pull/4660
   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   Use the exact first offset of message set when rolling log segment. This is possible to
do for message format V2 and beyond without any performance penalty, because we have the first
offset stored in the header. This augments the fix made in KAFKA-4451 to avoid using the heuristic
for V2 and beyond messages.
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   Added unit tests to simulate cases where segment needs to roll because of overflow in index
offsets. Verified that the new segment created in these cases uses the first offset, instead
of the heuristic in use previously.
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Use actual first offset of messages when rolling log segment for magic v2
> -------------------------------------------------------------------------
>                 Key: KAFKA-6530
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6530
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jason Gustafson
>            Assignee: Dhruvil Shah
>            Priority: Major
> We've implemented a heuristic to avoid overflowing when rolling a log segment to determine
the base offset of the next segment without decompressing the message set to find the actual
first offset. With the v2 message format, we can find the first offset without needing decompression,
so we can set the correct base offset exactly.

This message was sent by Atlassian JIRA

View raw message