Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <CAFc58G8Q1XVf+b7yBeqJW++J226VVEhrZR1W+S-WFF6hZNdDTA@mail.gmail.com>
References: <CAMd5Ysz4p6ndhZrdXN+AG3c6C32Y5dH2ZvTE0gT+3PrU1PuYkw@mail.gmail.com>
 <CAFc58G8Q1XVf+b7yBeqJW++J226VVEhrZR1W+S-WFF6hZNdDTA@mail.gmail.com>
From: Jun Rao <jun@confluent.io>
Date: Mon, 11 Dec 2017 13:31:18 -0800
Message-ID: <CAFc58G-qMoWMbyGFRJFteUGbd9CytA-a6XeEZZJukneHcdPiaQ@mail.gmail.com>
Subject: Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Content-Type: multipart/alternative; boundary="001a114717e246907705601741bd"
archived-at: Mon, 11 Dec 2017 21:33:55 -0000

--001a114717e246907705601741bd
Content-Type: text/plain; charset="UTF-8"

Another question is on the compatibility. Since now there are 2 ways of
specifying a partition reassignment, one under /admin/reassign_partitions
and the other under /admin/reassignments, we probably want to prevent the
same topic being reassigned under both paths at the same time?
Thanks,

Jun


On Fri, Dec 8, 2017 at 5:41 PM, Jun Rao <jun@confluent.io> wrote:

> Hi, Tom,
>
> Thanks for the KIP. It definitely addresses one of the pain points in
> partition reassignment. Another issue that it also addresses is the ZK node
> size limit when writing the reassignment JSON.
>
> My only concern is that the KIP needs to create one watcher per reassigned
> partition. This could add overhead in ZK and complexity for debugging when
> lots of partitions are being reassigned simultaneously. We could
> potentially improve this by introducing a separate ZK path for change
> notification as we do for configs. For example, every time we change the
> assignment for a set of partitions, we could further write a sequential
> node /admin/reassignment_changes/[change_x]. That way, the controller
> only needs to watch the change path. Once a change is triggered, the
> controller can read everything under /admin/reassignments/.
>
> Jun
>
>
> On Wed, Dec 6, 2017 at 1:19 PM, Tom Bentley <t.j.bentley@gmail.com> wrote:
>
>> Hi,
>>
>> This is still very new, but I wanted some quick feedback on a preliminary
>> KIP which could, I think, help with providing an AdminClient API for
>> partition reassignment.
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%
>> 3A+Interruptible+Partition+Reassignment
>>
>> I wasn't sure whether to start fleshing out a whole AdminClient API in
>> this
>> KIP (which would make it very big, and difficult to read), or whether to
>> break it down into smaller KIPs (which makes it easier to read and
>> implement in pieces, but harder to get a high-level picture of the
>> ultimate
>> destination). For now I've gone for a very small initial KIP, but I'm
>> happy
>> to sketch the bigger picture here if people are interested.
>>
>> Cheers,
>>
>> Tom
>>
>
>

--001a114717e246907705601741bd--