kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6347) Starting offset breach based log segment deletion never considers active segment
Date Tue, 12 Dec 2017 00:18:00 GMT
Guozhang Wang created KAFKA-6347:
------------------------------------

             Summary: Starting offset breach based log segment deletion never considers active
segment
                 Key: KAFKA-6347
                 URL: https://issues.apache.org/jira/browse/KAFKA-6347
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Guozhang Wang


This observation is related to KIP-107: in {{Log#deleteLogStartOffsetBreachedSegments}}, we
designed the predicate to enforce that the current segment can only be considered for deletion,
when it 1) has the next segment, 2) the next segment's starting offset is no larger than the
log start offset.

{code}
def shouldDelete(segment: LogSegment, nextSegmentOpt: Option[LogSegment]) =
      nextSegmentOpt.exists(_.baseOffset <= logStartOffset)
{code}

This means that, the current active segment would never be considered for purging. Normally
this is OK given that we will eventually roll out a new segment. However, with the default
size of segment of 1GB it means that with the purge data API we are in the worst case not
being able to purge the data for up to 1GB per topic partition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message