kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Becker <Thomas.Bec...@tivo.com>
Subject Re: [VOTE] KIP-349 Priorities for Source Topics
Date Mon, 08 Oct 2018 19:35:08 GMT
Well my (perhaps flawed) understanding of topic priorities is that lower priority topics are
not consumed as long as higher priority ones have unconsumed messages (which means our position
< HW). So if I'm doing this manually, I have to make some determination as to whether my
high priority topic partitions are at the HW before I can decide if I want to poll the lower
priority ones. Right?

On Fri, 2018-10-05 at 11:34 -0700, Colin McCabe wrote:

On Fri, Oct 5, 2018, at 10:58, Thomas Becker wrote:

Colin,

Would you mind sharing your vision for how this looks with multiple

consumers? I'm still getting my bearings with the new consumer but it's

not immediately obvious to me how this would work.


Hi Thomas,


I was just responding to the general idea that you would have some kind of control topic that
you wanted to read with very low latency, and some kind of set of data topics where the latency
requirements are less strict.  In that case, you can just have two consumers: one for the
low-latency topic, and one for the less low-latency topics.


There's a lot of things in this picture that are unclear.  Does the data in one set of topics
have any relation to the data in the other?  Why do we want a control channel distinct from
the data channel?  That's why I asked for clarification on the use-case.


In particular, it doesn't seem particularly easy to know when you are at the high

watermark of a topic.


KafkaConsumer#committed will return the last committed offset for a partition.  However, I'm
not sure I understand why you want this information in this case-- can you expand a bit on
this?


best,

Colin




-Tommy


On Mon, 2018-10-01 at 13:43 -0700, Colin McCabe wrote:


Hi all,



I feel like the DISCUSS thread didn't really come to a conclusion, so a

vote would be premature here.



In particular, I still don't really understand the use-case for this

feature.  Can someone give a concrete scenario where you would need

this?  The control plane / data plane example that is listed in the KIP

doesn't require this feature.  You can just have one consumer for the

control plane, and one for the data plane, and do priority that way.

The discussion feels kind of unfocused since we haven't identified even

one concrete use-case that needs this feature.



Unfortunately, this is a feature which consumes server-side memory.  We

have to store the priorities somehow when doing incremental fetch

requests.  If we go with an int as suggested, then this is at least 4

bytes per partition per incremental fetch request.  It also makes it

more complex and potentially slower to maintain the linked list of

partitions in the fetch requests.  Before we think about this, I'd like

to have a concrete use-case in mind, so that we can evaluate the costs

versus benefits.



best,


Colin




On Mon, Oct 1, 2018, at 07:47, Dongjin Lee wrote:


Great. +1 (non-binding)



On Mon, Oct 1, 2018 at 4:23 AM Matthias J. Sax

<matthias@confluent.io<mailto:matthias@confluent.io><mailto:matthias@confluent.io<mailto:matthias@confluent.io>>>


wrote:



+1 (binding)



As Dongjin pointed out, the community is working on upcoming 2.1


release, and thus it might take some time until people find time to


follow up on this an vote.




-Matthias



On 9/30/18 11:11 AM, nick@afshartous.com<mailto:nick@afshartous.com><mailto:nick@afshartous.com<mailto:nick@afshartous.com>>
wrote:



On Sep 30, 2018, at 5:16 AM, Dongjin Lee

<dongjin@apache.org<mailto:dongjin@apache.org><mailto:dongjin@apache.org<mailto:dongjin@apache.org>>>
wrote:



1. Your KIP document


<


https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics


<


https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics



lacks hyperlink to the discussion thread. And I couldn`t find the


discussion thread from the mailing archive.




Hi Dongjin,



There has been a discussion thread.  I added this link as a reference



  https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:kip-349


<https://lists.apache.org/list.html?dev@kafka.apache.org:lte=1M:kip-349>



to the KIP-349 page




https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics


<


https://cwiki.apache.org/confluence/display/KAFKA/KIP-349:+Priorities+for+Source+Topics




Best,


--


      Nick







--


*Dongjin Lee*



*A hitchhiker in the mathematical world.*



*github:  <http://goog_969573159/>github.com/dongjinleekr


<http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr


<http://kr.linkedin.com/in/dongjinleekr>slideshare:


www.slideshare.net/dongjinleekr<http://www.slideshare.net/dongjinleekr<http://www.slideshare.net/dongjinleekr<http://www.slideshare.net/dongjinleekr>>


<http://www.slideshare.net/dongjinleekr>*


________________________________


This email and any attachments may contain confidential and privileged

material for the sole use of the intended recipient. Any review,

copying, or distribution of this email (or any attachments) by others is

prohibited. If you are not the intended recipient, please contact the

sender immediately and permanently delete this email and any

attachments. No employee or agent of TiVo Inc. is authorized to conclude

any binding agreement on behalf of TiVo Inc. by email. Binding

agreements with TiVo Inc. may only be made by a signed written

agreement.

________________________________

This email and any attachments may contain confidential and privileged material for the sole
use of the intended recipient. Any review, copying, or distribution of this email (or any
attachments) by others is prohibited. If you are not the intended recipient, please contact
the sender immediately and permanently delete this email and any attachments. No employee
or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc.
by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message