kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dong Lin <lindon...@gmail.com>
Subject Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Date Thu, 03 Aug 2017 01:24:15 GMT
Hey Hu,

Actually, the solution suggested in the KIP would not work. A broker may
receive LeaderAndIsrRequest to create a large number of partitions when
user rebalance the load across brokers. All these partitions will be
created in the same log directory if the broker selects the log directory
with the most free space. The problem is that this may cause load imbalance
later as the size of those newly-created replicas will increase quickly.

One possible solution is to estimate the size of the replica based on the
average size of replicas in the broker. A more advanced and fine-granular
solution could estimate the expected free space based on the replica
retention configuration, the time a replica has been created, and the
current size of the replica. Can you think about it and maybe write down
the algorithm in more detail? And it may be worth thinking whether we can
simply replace the existing log directory selection algorithm without
adding a new config.

Thanks,
Dong

On Wed, Aug 2, 2017 at 6:10 PM, Dong Lin <lindong28@gmail.com> wrote:

> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <huxi_2b@hotmail.com> wrote:
>
>> Hi all, how do you think of this KIP? Any comments are welcomed.
>>
>>
>> ________________________________
>> 发件人: Hu Xi <huxi_2b@hotmail.com>
>> 发送时间: 2017年7月18日 15:21
>> 收件人: dev@kafka.apache.org
>> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
>>
>>
>> Hi all,
>>
>>      KIP-178 is created for a discussion on how LogManager selects log
>> directory. In this KIP, a new strategy is introduced to allow for the real
>> disk spaces for each directories. Be free to drop your comments here.
>> Thanks.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message