kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "huxihx (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-5560) LogManager should be able to create new logs based on free disk space
Date Wed, 19 Jul 2017 08:40:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

huxihx updated KAFKA-5560:
--------------------------
    Description: 
Currently, log manager chooses a directory configured in `log.dirs` by calculating the number
partitions in each directory and then choosing the one with the fewest partitions. But in
some real production scenarios where data volumes of partitions are not even, some disks nearly
become full whereas the others have a lot of spaces which lead to a poor data distribution.

We should offer a new strategy to users to have log manager honor the real disk free spaces
and choose the directory with the most disk space. Maybe a new broker configuration parameter
is needed, `log.directory.strategy` for instance. A new KIP is created to track this issue:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-178%3A+Size-based+log+directory+selection+strategy

Does it make sense?

  was:
Currently, log manager chooses a directory configured in `log.dirs` by calculating the number
partitions in each directory and then choosing the one with the fewest partitions. But in
some real production scenarios where data volumes of partitions are not even, some disks nearly
become full whereas the others have a lot of spaces which lead to a poor data distribution.

We should offer a new strategy to users to have log manager honor the real disk free spaces
and choose the directory with the most disk space. Maybe a new broker configuration parameter
is needed, `log.directory.strategy` for instance. Perhaps this needs a new KIP also.

Does it make sense?


> LogManager should be able to create new logs based on free disk space
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-5560
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5560
>             Project: Kafka
>          Issue Type: Improvement
>          Components: log
>    Affects Versions: 0.11.0.0
>            Reporter: huxihx
>              Labels: kips
>
> Currently, log manager chooses a directory configured in `log.dirs` by calculating the
number partitions in each directory and then choosing the one with the fewest partitions.
But in some real production scenarios where data volumes of partitions are not even, some
disks nearly become full whereas the others have a lot of spaces which lead to a poor data
distribution.
> We should offer a new strategy to users to have log manager honor the real disk free
spaces and choose the directory with the most disk space. Maybe a new broker configuration
parameter is needed, `log.directory.strategy` for instance. A new KIP is created to track
this issue: https://cwiki.apache.org/confluence/display/KAFKA/KIP-178%3A+Size-based+log+directory+selection+strategy
> Does it make sense?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message