kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject max bytes
Date Thu, 29 Mar 2012 08:51:29 GMT
Hey Guys,

The new FetchRequest in the 0.8 branch has a fetch_size per
topic/partition. Does this make sense? The purpose of the fetch_size
is to allow the client to control how much data they need to fetch at
once. I think setting this per-partition is a mistake since I don't
really consider how many partitions or topics there are. For example,
maybe there is a single partition on the server, so I configure 1MB
which is the most memory I want to allocate, then someone adds many
partitions on the server changing how much I fetch. Likewise based on
how the partition balancing comes out I will have larger or smaller
requests to a given server. Wouldn't it make more sense for there to
be a single size at the top level of the request controlling the total
size across all topic/partitions?

One detail is that to avoid "starvation" with a single size (i.e. if
the first partition could always satisfy that size) the client should
rotate the order of the topics and partitions in the fetch.

Thoughts?

-Jay

Mime
View raw message