kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient
Date Thu, 20 Jul 2017 15:47:41 GMT
Hi Tom,

Yes, a poll-based approach sounds good since reassignment can take a long
time. I even think that it should be manual. That is, the user should run a
command to ask for the status of the rebalance instead of the tool doing it
automatically.

Ismael

On Thu, Jul 20, 2017 at 4:12 AM, Tom Bentley <t.j.bentley@gmail.com> wrote:

> Hi Ismael,
>
> I've been working on the progress reporting assuming that it would be
> acceptable for the ReassignPartitionsCommand to poll the AdminClient API
> (and in turn the AdminClient API to poll the broker) in order to report
> progress in an interactive way.
>
> The alternative would, of course, be for the broker to push notify the
> AdminClient when it became aware of changes in progress. I didn't think
> this is what you meant as it would be a departure from the Kafka norm of
> request-response, but thought it worthwhile to check before I spend any
> more time on a polling-based approach.
>
> Thanks,
>
> Tom
>
>
>
> On 19 July 2017 at 16:08, Tom Bentley <t.j.bentley@gmail.com> wrote:
>
> > Ah, thank you! I took the number from the "Next KIP Number: 178" on the
> > KIP index and didn't check the tables. So this is now KIP-179. The old
> link
> > will point you to the right place.
> >
> > On 19 July 2017 at 15:55, Ismael Juma <ismael@juma.me.uk> wrote:
> >
> >> One more thing, it looks like there is already a KIP-178:
> >>
> >> KIP-178: Size-based log directory selection strategy
> >>
> >> Ismael
> >>
> >> On Wed, Jul 19, 2017 at 7:05 AM, Tom Bentley <t.j.bentley@gmail.com>
> >> wrote:
> >>
> >> > OK, I will work on adding support for this to the KIP, with the
> >> intention
> >> > of a two part implementation.
> >> >
> >> > On 19 July 2017 at 14:59, Ismael Juma <ismael@juma.me.uk> wrote:
> >> >
> >> > > Hi Tom,
> >> > >
> >> > > It's fine for the tool not to have this functionality from the
> start.
> >> > > However, since we're adding new Kafka protocol APIs, we need to
> >> consider
> >> > > some of these details to ensure we're building towards the end
> state,
> >> if
> >> > > that makes sense. Protocol APIs are used by multiple clients, so
> >> there is
> >> > > value in thinking ahead a bit when it comes to the design. The
> >> > > implementation can often be done in stages.
> >> > >
> >> > > Does that make sense?
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Wed, Jul 19, 2017 at 6:23 AM, Tom Bentley <t.j.bentley@gmail.com
> >
> >> > > wrote:
> >> > >
> >> > > > Hi Ismael,
> >> > > >
> >> > > > Answers in-line:
> >> > > >
> >> > > > 1. Have you considered how progress would be reported? Partition
> >> > > > > reassignment can take a long time and it would be good to
have a
> >> > > > mechanism
> >> > > > > for progress reporting.
> >> > > > >
> >> > > >
> >> > > > The ReassignPartitionsCommand doesn't currently have a mechanism
> to
> >> > track
> >> > > > progress. All you can do at the moment is initiate a reassignment
> >> (with
> >> > > > --execute), and later check whether the assignment is in the
state
> >> you
> >> > > > asked for (with --verify). I agree it would be nice to be able
to
> >> track
> >> > > > progress.
> >> > > >
> >> > > > This will be the first 'big' bit of work I've done on Kafka,
so I
> >> would
> >> > > > prefer to limit the scope of this KIP where possible. That said,
I
> >> > > suppose
> >> > > > it could be done by having receiving controllers publish their
> >> progress
> >> > > to
> >> > > > ZooKeeper, and adding Protocol and AdminClient API for getting
> this
> >> > > > information. If you're keen on this I can certainly modify the
KIP
> >> to
> >> > add
> >> > > > this.
> >> > > >
> >> > > > Alternatively I could write a second KIP to add this ability.
What
> >> > other
> >> > > > long running tasks are there for which we'd like the ability
to
> >> report
> >> > > > progress? If there are others it might be possible to come up
> with a
> >> > > common
> >> > > > mechanism.
> >> > > >
> >> > > >
> >> > > > > 2. Removals can only happen in major releases. In your example,
> >> the
> >> > > > removal
> >> > > > > could only happen in 2.0.0.
> >> > > > >
> >> > > >
> >> > > > OK, I'll update the KIP.
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message