kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhavesh Mistry <mistry.p.bhav...@gmail.com>
Subject [Java New Producer] Changing Partition number and its Impact
Date Tue, 04 Nov 2014 19:39:56 GMT
Hi Kafka Dev,

With new Producer, we are having to change the # partitions for a topic,
and we face this issue BufferExhaustedException.

Here is example,   we have set 64MiB and 32 partitions and 1MiB of batch
size.  But when we increase the partition to 128, it throws
BufferExhaustedException right way (non key based message).  Buffer is
allocated based on batch.size.  This is very common need to set auto
calculate batch size when partitions increase because we have about ~5000
boxes and it is not practical to deploy code in all machines than expand
partition for  scalability purpose.   What are options available while new
producer is running and partition needs to increase and not enough buffer
to allocate batch size for additional partition ?

buffer.memory=64MiB
batch.size=1MiB
block.on.buffer.full=false


Thanks,

Bhavesh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message