kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin McCabe" <cmcc...@apache.org>
Subject Re: [VOTE] KIP-349 Priorities for Source Topics
Date Tue, 15 Jan 2019 19:26:31 GMT
On Sun, Jan 13, 2019, at 18:13, nick@afshartous.com wrote:
> Thanks Colin and Mathias.
> > On Jan 12, 2019, at 8:27 PM, Matthias J. Sax <matthias@confluent.io> wrote:
> > 
> > Thus, I would suggest to limit this KIP to the consumer only, otherwise,
> > the scope will be too large and this KIP will drag on even longer. If we
> > really want to add this to Kafka Streams, I expect a long and difficult
> > discussion about this by itself, and thus, doing this in a follow up KIP
> > (if there is any demand) seems to be the better approach.
> > 
> Agreed, and my intent is to limit the scope to the consumer.  
> > About the starvation issue: maybe it's a bold claim, but is a potential
> > starvation of a low-priority topic not intended by design if topics have

Yeah, I was thinking this as well.  If you want strict priority behavior, high priority tasks
should always take priority over low priority ones.

> On reflection, it would be hard to describe the semantics of an API 
> that tried to address starvation by temporarily disabling 
> prioritization, and then oscillating back and forth. 
> Thus I agree that it makes sense not to try and address starvation to 
> Mathias’ point that this is intended by design.  The KIP has been 
> updated to reflect this by removing the second method.  

Yeah, I agree with that.  The problem is, the actual policy you want is kind of complex. 
It's probably better in the application rather than in Kafka.

> Regarding incremental fetch, Colin do you have any suggestion on which 
> option to adopt or how to proceed ?  

I think it makes sense to go back to use-cases again.  So far, all of the use-cases we discussed
could be handled by pause and resume.  So it makes sense to try to figure out what the issue
with those APIs is.  Are they not well-documented enough?  Is there something higher-level
we could build on top to make them easier to use?

It would be better to wait until a user comes forward and with a case where priorities are
needed, to implement them.  Since then we would know more about what the API should be, etc.


View raw message