kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dong Lin <lindon...@gmail.com>
Subject Re: [DISCUSS] KIP-286: producer.send() should not block on metadata update
Date Sat, 14 Apr 2018 06:34:08 GMT
Hey Becket,

Good point! Thanks for the comment.

I have updated the KIP to move the compression to user thread in the common
case. Basically user thread can be responsible for compressing and moving
messages from per-topic queue to per-partition queue once the metadata is
available. Only if IO threads has nothing to do (e.g. all messages in the
per-partition queue has been sent), then the IO thread can try to compress
and move some messages to the per-partition queue. Can you see if the
latest solution in the KIP address the problem?

Also, I have added a new section to analyze how the changes proposed in
this KIP would change the performance of producer.

Thanks,
Dong


On Fri, Apr 13, 2018 at 11:01 AM, Becket Qin <becket.qin@gmail.com> wrote:

> Thanks for the KIP, Dong.
>
> In the current threading model, compression is done by the user threads,
> therefore the producer sender thread can focus on IO. With the proposed
> changes, does that mean the producer sender thread will have to do all the
> compression as well? Would this become a performance bottleneck?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Apr 12, 2018 at 1:41 AM, Dong Lin <lindong28@gmail.com> wrote:
>
> > Hey Ted,
> >
> > Thanks for your comments. With the proposed solution in the KIP, the
> memory
> > is only allocated once for the given message, which is the same as the
> > existing implementation. The serialized message will be moved from
> > per-topic queue to per-partition queue without incurring additional
> memory
> > overhead. Does this address your question?
> >
> > Thanks,
> > Dong
> >
> >
> > On Wed, Apr 11, 2018 at 9:36 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Looks like per-topic queue is introduced.
> > >
> > > In terms of memory consumption, how does the KIP allocate memory
> > > between per-topic
> > > queue and per-partition queue ?
> > >
> > > Thanks
> > >
> > > On Wed, Apr 11, 2018 at 8:50 PM, Dong Lin <lindong28@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created KIP-286: producer.send() should not block on metadata
> > > > update. See
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 286%3A+producer.send%28%29+should+not+block+on+metadata+update
> > > > .
> > > >
> > > > The KIP intends to improve user-experience of producer.send() when
> > > metadata
> > > > is not available. It is related but different from the previous
> > > discussion
> > > > in KAFKA-3539 in the sense that user still has the option of letting
> > > > producer.send() block on full producer queue.
> > > >
> > > > Comments are welcome!
> > > >
> > > > Thanks,
> > > > Dong
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message