samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Feng <fengta...@gmail.com>
Subject Re: Use one producer for both coordinator stream and users system?
Date Tue, 18 Aug 2015 17:15:45 GMT
Thanks Yan. I guess I am not very clear with the coordinatorStream concept
before.
-Tao

On Tue, Aug 18, 2015 at 12:26 AM, Yan Fang <yanfang724@gmail.com> wrote:

> Hi Tao,
>
> First, one kafka producer has an i/o thread. (correct me if I am wrong).
>
> Second, after Samza 0.10.0, we have a coordinator stream, which stores the
> checkpoint, config and other locality information for auto-scaling, dynamic
> configuration, etc purpose. (See Samza-348
> <https://issues.apache.org/jira/browse/SAMZA-348>). So we have a producer
> for this coordinator stream.
>
> Therefore, each contains will have at least two producers, one is for the
> coordinator stream, one is for the users system.
>
> My question is, can we use only one producer for both coordinator stream
> and the users system to have better performance? (from the doc, it may
> retrieve better performance.)
>
> Thanks,
>
> Fang, Yan
> yanfang724@gmail.com
>
> On Mon, Aug 17, 2015 at 9:49 PM, Tao Feng <fengtao04@gmail.com> wrote:
>
> > Hi Yan,
> >
> > Naive question: what do we need producer thread of coordinator stream
> for?
> >
> > Thanks,
> > -Tao
> >
> > On Mon, Aug 17, 2015 at 2:09 PM, Yan Fang <yanfang724@gmail.com> wrote:
> >
> > > Hi guys,
> > >
> > > I have this question because Kafka's doc
> > > <
> > >
> >
> http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
> > > >
> > > seems recommending having one producer shared by all threads ("*The
> > > producer is thread safe and should generally be shared among all
> threads
> > > for best performance.*"), while currently the coordinator stream is
> > using a
> > > separate producer (usually, there are two producers(two producer
> threads)
> > > in each container: one is for the coordinator stream , one is for the
> > > "real" job)
> > >
> > > 1. Will having one producer shared by all thread really improve the
> > > performance? (haven't done the perf test myself. Guess Kafka has some
> > > proof).
> > >
> > > 2. if yes, should we go this way?
> > >
> > > Thanks,
> > >
> > > Fang, Yan
> > > yanfang724@gmail.com
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message