kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient
Date Tue, 25 Jul 2017 10:38:39 GMT
Hi Tom and Paolo,

It's true that increasing the number of partitions is done via the
kafka-topics tool, which is also being converted to use the AdminClient
(but via a different JIRA). I also agree that it would be good to consider
if alterTopics would be a sensible way to support all the use cases or if
it's better to have separate APIs. I think it makes sense to have a single
KIP though as they are related and it will be easier to evaluate as a whole.

Does that make sense?

Ismael

On Tue, Jul 25, 2017 at 10:16 AM, Paolo Patierno <ppatierno@live.com> wrote:

> Hi,
>
>
> I was digging into it because I need something like an Admin Client alter
> API for my work on rewriting the TopicCommand tool using them.
>
> The AlterConfigs API is used for changing topic level configuration (i.e.
> retention.ms, retention.bytes and so on).
>
> A new AlterTopic API could be better in order to change topic "high level"
> structure so number of partitions, replication factors and so on.
>
> My opinion is that we need separate API because from my point of view they
> are different settings.
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>
>
> ________________________________
> From: Tom Bentley <t.j.bentley@gmail.com>
> Sent: Tuesday, July 25, 2017 9:02 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use
> AdminClient
>
> Hi Ismael,
>
> I assume that's the --partitions for kafka-topics.sh? I must admit I hadn't
> considered that tool, only kafka-reassign-partitions.sh. Thanks for
> pointing it out, because obviously the AdminClient API needs to be suitable
> for reuse in kafka-topics.sh too. Since AdminClient doesn't currently
> support an alterTopics() I guess the API should mirror the API for
> newTopics(), so something like:
>
>     public AlterTopicsResult alterTopics(Set<AlteredTopic> topics);
>     // where:
>     public class AlteredTopic {
>         // constructor etc
>         public String name();
>         public int numPartitions();
>         public int replicationFactor();
>         Map<Integer,List<Integer>> replicasAssignment();
>     }
>
> Note that although NewTopic contains a `Map<String,String> configs`, I
> think the API for changing a topic's config already exists: alterConfigs().
>
> This API is better than having separate methods to set the number of
> partitions/replicas and assigning them to brokers, since sometimes people
> will want to set the assignment at the same time as changing the
> partitions/replicas.
>
> An API like this could then be used by both tools.
>
>
>
> On 24 July 2017 at 16:23, Ismael Juma <ismael@juma.me.uk> wrote:
>
> > Hi Tom,
> >
> > I think it makes sense to keep progress reporting simple in the initial
> > version. As you say, time to completion is tricky to compute and it seems
> > like it should be tackled as its own KIP.
> >
> > Regarding waiting for reassignment completion, it's easy enough for
> people
> > to do that via a script, so I don't think we need to add it to the tool.
> >
> > One question that occurred to me, the tool allows one to add partitions
> to
> > existing topics:
> >
> > val partitionsOpt = parser.accepts("partitions", "The number of
> partitions
> > for the topic being created or " +
> >       "altered (WARNING: If partitions are increased for a topic that
> has a
> > key, the partition logic or ordering of the messages will be affected")
> >
> > It seems like it may make sense to have that as an AdminClient API as
> well.
> > If we agree to do that, then we need to decide if it should be
> implemented
> > client-side or by adding a protocol API. The former is simpler, but the
> > latter would allow non Java clients to use it without duplicating the
> logic
> > of assigning replicas to the new partitions. What are your thoughts?
> >
> > Ismael
> >
> > On Sat, Jul 22, 2017 at 10:14 AM, Tom Bentley <t.j.bentley@gmail.com>
> > wrote:
> >
> > > Thinking about this some more, I release that the proposed API for
> > tracking
> > > progress is a bit specific to this reassignment use case. A more
> > generally
> > > useful API would be to be able to find out, for a partition on a
> broker:
> > >
> > > * When the broker became a follower
> > > * When the broker was last in the ISR
> > > * If the broker is not in the ISR, how many messages behind it is
> > >
> > > That's not enough to be able to calculate a percentage completion of a
> > > reassignment though (we don't know how far behind it was when it
> became a
> > > follower). But maybe we could maintain a maximum of how far behind it
> has
> > > fallen behind the ISR, since it became a follower/dropped out of the
> ISR.
> > >
> > > btw, we need the middle bullet to cope with following sequence:
> > >
> > > 1. Start syncing
> > > 2. Finish syncing
> > > 3. Fall behind (drop out of ISR for some reason)
> > > 4. User queries for if reassignment has finished (it has, but just
> > looking
> > > at the ISR would give the impression that it has not).
> > >
> > >
> > >
> > >
> > > On 21 July 2017 at 11:09, Tom Bentley <t.j.bentley@gmail.com> wrote:
> > >
> > > > Aside: I've started this new DISCUSS thread for KIP-179 since the
> > > original
> > > > one had the incorrect KIP number 178. The original thread can be
> found
> > > > here: http://mail-archives.apache.org/mod_mbox/kafka-dev/201707.
> > > > mbox/%3cCAMd5YszudP+-8z5KTbFh6JscT2p4xFi1=VZWWX+
> > > > 5DccPxRyavw@mail.gmail.com%3e
> > > >
> > > > I've just updated KIP-179 to support Ismael's request for the command
> > to
> > > > be able to support progress reporting of an ongoing partition
> > > reassignment.
> > > >
> > > > I'll call out two things which I'm not sure about since I don't yet
> > have
> > > > much experience of Kafka being used operationally:
> > > >
> > > > 1. As currently constructed the --progress option could report an
> > overall
> > > > progress percentage, per-partition percentages and errors. It cannot
> > > > provide any kind of time-to-completion estimate. Part of me is loath
> to
> > > do
> > > > this, as I'm sure we all remember file transfer dialogs that provide
> > > > amusing/baffling time-to-completion estimates. So it might be hard to
> > do
> > > > _well_. On the other hand I expect the thing people will be
> interested
> > in
> > > > will often be "when will it be finished?"
> > > >
> > > > 2. There is no option for the tool to wait for reassignment
> > completion. I
> > > > can imagine users might want to script something to happen after the
> > > > reassignment is complete, and without some kind of --wait option they
> > > will
> > > > have to poll for completion "manually". Having a --wait optin and
> > putting
> > > > this polling in the tool means we have a lot more control over how
> > often
> > > > such polling happens.
> > > >
> > > > The KIP is available here:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+
> > > > ReassignPartitionsCommand+to+use+AdminClient
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message