kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: max bytes
Date Thu, 29 Mar 2012 16:49:25 GMT
Here is another thought. An alternative is to take the fetch.size in
consumer config and divide it among all partitions. Initially, this can be
just simple even distribution. Later on, we can be a bit smarter and give
larger partitions larger fetch size.

Thanks,

Jun

On Thu, Mar 29, 2012 at 1:51 AM, Jay Kreps <jay.kreps@gmail.com> wrote:

> Hey Guys,
>
> The new FetchRequest in the 0.8 branch has a fetch_size per
> topic/partition. Does this make sense? The purpose of the fetch_size
> is to allow the client to control how much data they need to fetch at
> once. I think setting this per-partition is a mistake since I don't
> really consider how many partitions or topics there are. For example,
> maybe there is a single partition on the server, so I configure 1MB
> which is the most memory I want to allocate, then someone adds many
> partitions on the server changing how much I fetch. Likewise based on
> how the partition balancing comes out I will have larger or smaller
> requests to a given server. Wouldn't it make more sense for there to
> be a single size at the top level of the request controlling the total
> size across all topic/partitions?
>
> One detail is that to avoid "starvation" with a single size (i.e. if
> the first partition could always satisfy that size) the client should
> rotate the order of the topics and partitions in the fetch.
>
> Thoughts?
>
> -Jay
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message