Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
From: Florian Hussonnois <fhussonnois@gmail.com>
Date: Thu, 1 Sep 2016 18:00:22 +0200
Message-ID: <CADu8QrnkMjQRDbLCpO8KewcWNcOsh0b4rJHFabL230ET8nM_4w@mail.gmail.com>
Subject: PartitionAssignor / Sort members per subscription time before
 assigning partitions
To: dev@kafka.apache.org
Content-Type: multipart/alternative; boundary=001a114b2ef4c42358053b744f50
archived-at: Thu, 01 Sep 2016 16:00:30 -0000

--001a114b2ef4c42358053b744f50
Content-Type: text/plain; charset=UTF-8

Hi Kafka Team,

I would like to have your opinion before creating a new JIRA.

I'm working with the Java Consumer API. The current partition assignors use
the consumer ids to sort members before assigning partitions.

This works pretty well as long as all consumers are started into the same
JVM and no more consumers than the number of partitions are created.

However, in many cases consumers are distributed across multiple hosts. To
continue consuming with an optimal number of consumers (even with a host
failure) we can create as many consumers as partitions on each host.

For example, we have two consumers C0, C1 each on a dedicated host and one
topic with 4 partitions. With current assignors it is not possible to have
2 consuming threads and 2 idle threads per host.

Instead of that, C0 will have 4 consuming threads and C1 will have 4 idle
threads.

One solution could be to keep a timestamp the first time a member
subscribes to a topic. This timestamp can then be used to sort members for
the partitions assignment. In this way, the partition assignment will be
more predictable as it will not depend on member ids.

One drawback of this solution is that the consumer responsible of
assignments will keep a local state.

Thanks,

-- 
Florian

--001a114b2ef4c42358053b744f50--