kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dong Lin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-6172) Cache lastEntry in TimeIndex to avoid unnecessary disk access
Date Mon, 06 Nov 2017 01:19:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dong Lin updated KAFKA-6172:
----------------------------
    Fix Version/s: 1.0.1

> Cache lastEntry in TimeIndex to avoid unnecessary disk access
> -------------------------------------------------------------
>
>                 Key: KAFKA-6172
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6172
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Dong Lin
>            Assignee: Dong Lin
>             Fix For: 1.0.1
>
>
> LogSegment.close() calls timeIndex.maybeAppend(...), which in turns make a number of
calls to timeIndex.lastEntry(). Currently timeIndex.lastEntry() involves disk seek operation
because it tries to read the content of the last few bytes of the index files on the disk.
This slows down the broker shutdown process.
> Here is the time of LogManager.shutdown() in various settings. In all these tests, broker
has roughly 6k partitions and 20k segments.
> - If broker does not have this patch and `log.dirs` is configured with 1 JBOD log directory,
LogManager.shutdown() takes 15 minutes (roughly 900 seconds).
> - If broker does not have this patch and `log.dirs` is configured with 10 JBOD log directories,
LogManager.shutdown() takes 84 seconds.
> - If broker have this patch and `log.dirs` is configured with 10 JBOD log directories,
LogManager.shutdown() takes 24 seconds.
> Thus we expect to save 71% time in LogManager.shutdown() by having this optimization.
This patch intends to reduce the broker shutdown time by caching the lastEntry in memory so
that broker does not have to always read disk to get the lastEntry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message