kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6346) Consolidate multiple background async log operations
Date Mon, 11 Dec 2017 23:10:00 GMT
Guozhang Wang created KAFKA-6346:

             Summary: Consolidate multiple background async log operations
                 Key: KAFKA-6346
                 URL: https://issues.apache.org/jira/browse/KAFKA-6346
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Guozhang Wang

On Kafka broker's log managers, today we trigger a lot of repeating and one-time async background
tasks, including:

1. log deletion based on retention policy (repeating).
  1.2. async log segment deletion triggered from 1 above (one-time)
2. log deletion based on replica stopping / log directory swapping (repeating)
3. log compaction, i.e. so-called "log cleaner thread" (repeating)
4. dirty log flushing
5. log recovery offsets checkpointing.
6. log starting offsets checkpointing.

I suspect all these background tasks are silently eating our CPU idle time unnecessarily.
We should consider 1) benchmark their impacts on CPU cost and 2) if impact is high, consider
consolidating some of these tasks into fewer async background threads.

This message was sent by Atlassian JIRA

View raw message