kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justine Olshan <jols...@confluent.io>
Subject Re: [DISCUSS] KIP-480 : Sticky Partitioner
Date Wed, 10 Jul 2019 15:48:28 GMT
Hi M,

I'm a little confused by what you mean by extending the behavior on to the
RoundRobinPartitioner.
The sticky partitioner plans to remove the round-robin behavior from
records with no keys. Instead of sending them to each partition in order,
it sends them all to the same partition until the batch is sent.
I don't think you can have both round-robin and sticky partition behavior.

Thank you,
Justine Olshan

On Wed, Jul 10, 2019 at 1:54 AM M. Manna <manmedia@gmail.com> wrote:

> Thanks for the comments Colin.
>
> My only concern is that this KIP is addressing a good feature and having
> that extended to RoundRobinPartitioner means 1 less KIP in the future.
>
> Would it be appropriate to extend the support to RoundRobinPartitioner too?
>
> Thanks,
>
> On Tue, 9 Jul 2019 at 17:24, Colin McCabe <cmccabe@apache.org> wrote:
>
> > Hi M,
> >
> > The RoundRobinPartitioner added by KIP-369 doesn't interact with this
> > KIP.  If you configure your producer to use RoundRobinPartitioner, then
> the
> > DefaultPartitioner will not be used.  And the "sticky" behavior is
> > implemented only in the DefaultPartitioner.
> >
> > regards,
> > Colin
> >
> >
> > On Tue, Jul 9, 2019, at 05:12, M. Manna wrote:
> > > Hello Justine,
> > >
> > > I have one item I wanted to discuss.
> > >
> > > We are currently in review stage for KAFKA-3333 where we can choose
> > always
> > > RoundRobin regardless of null/usable key.
> > >
> > > If I understood this KIP motivation correctly, you are still honouring
> > how
> > > the hashing of key works for DefaultPartitioner. Would you say that
> > having
> > > an always "Round-Robin" partitioning with "Sticky" assignment
> (efficient
> > > batching of records for a partition) doesn't deviate from your original
> > > intention?
> > >
> > > Thanks,
> > >
> > > On Tue, 9 Jul 2019 at 01:00, Justine Olshan <jolshan@confluent.io>
> > wrote:
> > >
> > > > Hello all,
> > > >
> > > > If there are no more comments or concerns, I would like to start the
> > vote
> > > > on this tomorrow afternoon.
> > > >
> > > > However, if there are still topics to discuss, feel free to bring
> them
> > up
> > > > now.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Tue, Jul 2, 2019 at 4:25 PM Justine Olshan <jolshan@confluent.io>
> > > > wrote:
> > > >
> > > > > Hello again,
> > > > >
> > > > > Another update to the interface has been made to the KIP.
> > > > > Please let me know if you have any feedback!
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jun 28, 2019 at 2:52 PM Justine Olshan <
> jolshan@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > >> Hi all,
> > > > >> I made some changes to the KIP.
> > > > >> The idea is to clean up the code, make behavior more explicit,
> > provide
> > > > >> more flexibility, and to keep default behavior the same.
> > > > >>
> > > > >> Now we will change the partition in onNewBatch, and specify the
> > > > >> conditions for this function call (non-keyed values, no explicit
> > > > >> partitions) in willCallOnNewBatch.
> > > > >> This clears up some of the issues with the implementation. I'm
> > happy to
> > > > >> hear further opinions and discuss this change!
> > > > >>
> > > > >> Thank you,
> > > > >> Justine
> > > > >>
> > > > >> On Thu, Jun 27, 2019 at 2:53 PM Colin McCabe <cmccabe@apache.org>
> > > > wrote:
> > > > >>
> > > > >>> On Thu, Jun 27, 2019, at 01:31, Ismael Juma wrote:
> > > > >>> > Thanks for the KIP Justine. It looks pretty good. A
few
> comments:
> > > > >>> >
> > > > >>> > 1. Should we favor partitions that are not under replicated?
> > This is
> > > > >>> > something that Netflix did too.
> > > > >>>
> > > > >>> This seems like it could lead to cascading failures, right?
 If a
> > > > >>> partition becomes under-replicated because there is too much
> > traffic,
> > > > the
> > > > >>> producer stops sending to it, which puts even more load on
the
> > > > remaining
> > > > >>> partitions, which are even more likely to fail then, etc.
 It
> also
> > will
> > > > >>> create unbalanced load patterns on the consumers.
> > > > >>>
> > > > >>> >
> > > > >>> > 2. If there's no measurable performance difference,
I agree
> with
> > > > >>> Stanislav
> > > > >>> > that Optional would be better than Integer.
> > > > >>> >
> > > > >>> > 3. We should include the javadoc for the newly introduced
> method
> > that
> > > > >>> > specifies it and its parameters. In particular, it would
good
> to
> > > > >>> specify if
> > > > >>> > it gets called when an explicit partition id has been
provided.
> > > > >>>
> > > > >>> Agreed.
> > > > >>>
> > > > >>> best,
> > > > >>> Colin
> > > > >>>
> > > > >>> >
> > > > >>> > Ismael
> > > > >>> >
> > > > >>> > On Mon, Jun 24, 2019, 2:04 PM Justine Olshan <
> > jolshan@confluent.io>
> > > > >>> wrote:
> > > > >>> >
> > > > >>> > > Hello,
> > > > >>> > > This is the discussion thread for KIP-480: Sticky
> Partitioner.
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner
> > > > >>> > >
> > > > >>> > > Thank you,
> > > > >>> > > Justine Olshan
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message