kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Bentley <t.j.bent...@gmail.com>
Subject Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient
Date Wed, 02 Aug 2017 12:12:01 GMT
Hi again Ismael,


1. It's worth emphasising that reassigning partitions is a different
>> process than what happens when a topic is created, so not sure trying to
>> make it symmetric is beneficial. In addition to what was already
>> discussed,
>> one should also enable replication throttling before moving the data.
>
>
> Thanks for your responses.The only advantage I can see from having
> separate APIs for partition count change vs. the other changes is that we
> expect the former to return synchronously, and the latter to (usually)
> return asynchronously. Lumping them into the same API forces clients of
> that API to code for the two possible cases. Is this the particular concern
> you had in mind, or do you have others?
>

Just to help see what this looks like I've created another version of
KIP-179 (
https://cwiki.apache.org/confluence/display/KAFKA/Copy+of+KIP-179+-+Change+ReassignPartitionsCommand+to+use+AdminClient
).

The AdminClient APIs are more type safe (harder to misuse). The
asynchronous nature of the alterReplicationFactors() and
reassignPartitions() methods aren't really apparent (you'd still have to
read the javadocs to know you had to call replicaStatus() to determine
completion), but it's still better that the lumped-together API because the
methods are _consistently_ async. The Protocol APIs are slightly nicer too,
in that INVALID_REQUEST is less commonly returned. Another benefit is it
would allow for these different alteration APIs to be modified
independently in the future, should that necessary.

What do others think?

You're right that the proposal doesn't cover setting and clearing a
> throttle, and it should. I will study the code about how this works before
> posting back about that.
>

AFAICS from the code the throttle is simply a dynamic config of the broker.
We already have the AdminClient.alterConfigs API for setting this, so the
refactoring of kafka-reassign-partitions.sh could just use that existing
API. We could still achieve your objective of setting the throttle before
reassignment by making the --throttle option mandatory when --execute was
present. Or did you have something else in mind?

Thanks,

Tom

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message