kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Bellemare <adam.bellem...@gmail.com>
Subject Re: [VOTE] KIP-349 Priorities for Source Topics
Date Fri, 11 Jan 2019 15:00:36 GMT
Hi Colin

Thanks for the sober second thought - I actually didn't see the
inconclusive parts of the DISCUSS (I must have missed them when going
through) so I am grateful you highlighted these. I will have to remove my
+1 in light of the issues Colin has mentioned, but I will follow the
discussion more carefully.



On Thu, Jan 10, 2019 at 5:41 PM Colin McCabe <cmccabe@apache.org> wrote:

> Hi all,
>
> Just as a quick reminder, this is not really a complete proposal.  There
> are a bunch of unresolved issues with this KIP.  One example is how this
> interacts with incremental fetch sessions.  It is not mentioned anywhere in
> the KIP text.  Previously we discussed some approaches, but there was no
> clear consensus.
>
> Another example is the issue of starvation.  The KIP discusses "an idea"
> for handling starvation, but the details are very sparse-- just a sentence
> of two.  At minimum we would need some kind of configuration for the
> proposed "lag deltas".  It's also not clear that the proposed mechanism
> would work, since we don't receive lag metrics for partitions that we don't
> fetch.  But if we do fetch from the partitions, we may receive data, which
> would cause our policy to not be strict prioties.  Keep in mind, even
> attempting to fetch 1 byte may cause us to read an entire message, as
> described in KIP-74.
>
> It seems that we don't understand the potential use-cases.  The only
> use-case referenced by the KIP is this one, by Bala Prassanna:
>
>  > We use Kafka to process the asynchronous events of our Document
> Management
>  > System such as preview generation, indexing for search etc.
>  > The traffic gets generated via Web and Desktop Sync application. In
> such
>  > cases, we had to prioritize the traffic from web and consume them
> first.
>  > But this might lead to the starvation of events from sync if the
> consumer
>  > speed is slow and the event rate is high from web.  A solution to
> handle
>  > the starvation with a timeout after which the events are consumed
> normally
>  > for a specified period of time would be great and help us use our
>  > resources effectively.
>
> Reading this carefully, it seems that the problem is actually starvation,
> not implementing priorities.  Bala already implemented priorities outside
> of Kafka.  If you read the discussion on KAFKA-6690, Bala also makes this
> comment: "We would need this in both Consumer API and Streams API."  The
> current KIP does not discuss adding priorities to Streams-- only to the
> basic consumer API.  So it seems clear that KIP-349 does not address Bala's
> use-case at all.
>
> Stepping back a little bit, it seems like a few people have spoken up
> recently asking for some way to re-order the messages they receive from the
> Kafka consumer.  For example, ChienHsing Wu has discussed a use-case where
> he wants to receive messages in a "round robin" order.  All of this is
> possible by doing some local buffering and using the pause and resume
> APIs.  Perhaps we should consider better documenting these APIs, and adding
> some examples.  Or perhaps we should consider some kind of API to do
> pluggable buffering on the client side.
>
> In any case, this needs more discussion.  We need to be clear and definite
> about what use cases we want to solve, and the tradeoffs we're making to
> solve them.  For now, I have to reiterate my -1 (binding).
>
> Colin
>
>
> On Thu, Jan 10, 2019, at 10:46, Adam Bellemare wrote:
> > Looks good to me then!
> >
> > +1 non-binding
> >
> >
> >
> > > On Jan 10, 2019, at 1:22 PM, Afshartous, Nick <nafshartous@wbgames.com>
> wrote:
> > >
> > >
> > > Hi Adam,
> > >
> > >
> > > This change is only intended for the basic consumer API.
> > >
> > >
> > > Cheers,
> > >
> > > --
> > >
> > >    Nick
> > >
> > >
> > > ________________________________
> > > From: Adam Bellemare <adam.bellemare@gmail.com>
> > > Sent: Sunday, January 6, 2019 11:45 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-349 Priorities for Source Topics
> > >
> > > Hi Nick
> > >
> > > Is this change only for the basic consumer? How would this affect
> anything with Kafka Streams?
> > >
> > > Thanks
> > >
> > >
> > >> On Jan 5, 2019, at 10:52 PM, nick@afshartous.com wrote:
> > >>
> > >> Bumping again for more votes.
> > >> --
> > >>     Nick
> > >>
> > >>
> > >>> On Dec 26, 2018, at 12:36 PM, nick@afshartous.com wrote:
> > >>>
> > >>> Bumping this thread for more votes
> > >>>
> > >>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e=
> ><
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D349-3A-2BPriorities-2Bfor-2BSource-2BTopics&d=DwIFAg&c=-SicqtCl7ffNuxX6bdsSog&r=P28z_ShLjFv5AP-w9-b_auYBx8qTrjk2JPYZKbjmJTs&m=5qg4fCOVMtRYYLu2e8h8KmDyis_uk3aFqT5Eq0x4hN8&s=Sbrd5XSwEZiMc9iTPJjRQafl4ubXwIOnsnFzhBEa0h0&e=
> >>
> > >>
> > >>
> > >>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message