kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: [VOTE] KIP-349 Priorities for Source Topics
Date Sun, 12 Aug 2018 17:53:59 GMT
Both use-case: Command queue for a streams job and work-queue in general
seem reasonable. Thank you for explaining.

On Sun, Aug 12, 2018 at 6:34 AM, Matt Farmer <matt@frmr.me> wrote:

> The work-queue use case is mostly how I see this being used, yes.
>
> In the most generic sense I can see its use in a situation where the
> business dictates
> that we have to guarantee quality of service for some set of low number of
> messages while
> there's some high number of messages being processed from a different
> topic.
>
> For our pipelines at work, this would actually make it possible for us to
> define a command
> and control topic for some of our streams applications. We occasionally
> have to change the
> behavior of our streams in reaction to system issues or, occasionally,
> user's abusing the
> system and creating a bunch of garbage data that we'd like to skip over.
> Today, we have
> to either define that behavior in a config setting and restart the
> application (which is what
> we currently do) or implement some sort of API external to streams that the
> stream pulls
> state from.
>
> With a CnC topic, we could interleave these into the normal stream
> processing flow and
> instruct it to alter a state store, for example, with the criteria for
> records to be dropped
> without introducing other libraries or having to manually synchronize with
> external state.
>
> On Wed, Aug 8, 2018 at 10:11 PM Gwen Shapira <gwen@confluent.io> wrote:
>
> > Can you guys spell it out for me? I just don't really see when I want to
> > subscribe to two topics but not get events from both at the same time.
> > Is this a work-queue type pattern?
> >
> > On Wed, Aug 8, 2018 at 6:10 PM, Matt Farmer <matt@frmr.me> wrote:
> >
> > > Oh, almost forgot, thanks for the KIP - I can see this being a very
> > useful
> > > addition. :)
> > >
> > > On Wed, Aug 8, 2018 at 9:09 PM Matt Farmer <matt@frmr.me> wrote:
> > >
> > > > Is it worth spelling out explicitly what the behavior is when two
> > topics
> > > > have the same priority? I'm a bit fuzzy on how we choose what topics
> to
> > > > consume from right now, if I'm being honest, so it could be useful to
> > > > outline the current behavior in the background and to spell out how
> > that
> > > > would change (or if it would change) when two topics are given the
> same
> > > > priority.
> > > >
> > > > Also, how does this play with max.poll.records? Does the consumer
> read
> > > > from all the topics in priority order until we've hit the number of
> > > records
> > > > or the poll timeout? Or does it immediately return the high priority
> > > > records without pulling low priority records?
> > > >
> > > > On Wed, Aug 8, 2018 at 8:39 PM <nick@afshartous.com> wrote:
> > > >
> > > >>
> > > >> Hi All,
> > > >>
> > > >> Calling for a vote on KIP-349
> > > >>
> > > >>
> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 349%3A+Priorities+for+Source+Topics
> > > >> <
> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 349:+Priorities+for+Source+Topics
> > > >> >
> > > >>
> > > >> Cheers,
> > > >> --
> > > >>       Nick
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > <http://www.confluent.io/blog>
> >
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message