kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephane Maarek <steph...@simplemachines.com.au>
Subject Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion
Date Wed, 28 Feb 2018 07:49:05 GMT
Sounds awesome !
Are you planning to have auto scaling of partitions in a following KIP ?
That would be the holy grail

On 28 Feb. 2018 5:13 pm, "Dong Lin" <lindong28@gmail.com> wrote:

> Hey Jan,
>
> I am not sure if it is acceptable for producer to be stopped for a while,
> particularly for online application which requires low latency. I am also
> not sure how consumers can switch to a new topic. Does user application
> needs to explicitly specify a different topic for producer/consumer to
> subscribe to? It will be helpful for discussion if you can provide more
> detail on the interface change for this solution.
>
> Thanks,
> Dong
>
> On Mon, Feb 26, 2018 at 12:48 AM, Jan Filipiak <Jan.Filipiak@trivago.com>
> wrote:
>
> > Hi,
> >
> > just want to throw my though in. In general the functionality is very
> > usefull, we should though not try to find the architecture to hard while
> > implementing.
> >
> > The manual steps would be to
> >
> > create a new topic
> > the mirrormake from the new old topic to the new topic
> > wait for mirror making to catch up.
> > then put the consumers onto the new topic
> >     (having mirrormaker spit out a mapping from old offsets to new
> offsets:
> >         if topic is increased by factor X there is gonna be a clean
> > mapping from 1 offset in the old topic to X offsets in the new topic,
> >         if there is no factor then there is no chance to generate a
> > mapping that can be reasonable used for continuing)
> >     make consumers stop at appropriate points and continue consumption
> > with offsets from the mapping.
> > have the producers stop for a minimal time.
> > wait for mirrormaker to finish
> > let producer produce with the new metadata.
> >
> >
> > Instead of implementing the approach suggest in the KIP which will leave
> > log compacted topic completely crumbled and unusable.
> > I would much rather try to build infrastructure to support the mentioned
> > above operations more smoothly.
> > Especially having producers stop and use another topic is difficult and
> > it would be nice if one can trigger "invalid metadata" exceptions for
> them
> > and
> > if one could give topics aliases so that their produces with the old
> topic
> > will arrive in the new topic.
> >
> > The downsides are obvious I guess ( having the same data twice for the
> > transition period, but kafka tends to scale well with datasize). So its a
> > nicer fit into the architecture.
> >
> > I further want to argument that the functionality by the KIP can
> > completely be implementing in "userland" with a custom partitioner that
> > handles the transition as needed. I would appreciate if someone could
> point
> > out what a custom partitioner couldn't handle in this case?
> >
> > With the above approach, shrinking a topic becomes the same steps.
> Without
> > loosing keys in the discontinued partitions.
> >
> > Would love to hear what everyone thinks.
> >
> > Best Jan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 11.02.2018 00:35, Dong Lin wrote:
> >
> >> Hi all,
> >>
> >> I have created KIP-253: Support in-order message delivery with partition
> >> expansion. See
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-253%
> >> 3A+Support+in-order+message+delivery+with+partition+expansion
> >> .
> >>
> >> This KIP provides a way to allow messages of the same key from the same
> >> producer to be consumed in the same order they are produced even if we
> >> expand partition of the topic.
> >>
> >> Thanks,
> >> Dong
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message