samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Feng <fengta...@gmail.com>
Subject Re: Measuring Samza Job Throughput
Date Thu, 18 Jun 2015 20:45:55 GMT
Hi, Milinda, Yi,

Sure. I will be happy to help on this.

Thanks,
-Tao

On Wed, Jun 17, 2015 at 11:35 AM, Yi Pan <nickpan47@gmail.com> wrote:

> Hi, Milinda,
>
> Tao @LinkedIn has done some Samza benchmark test using a standard
> word-count task. You may want to reach out to him for some detailed ideas
> on how to set up the perf tests.
>
> Best!
>
> -Yi
>
> On Wed, Jun 17, 2015 at 11:25 AM, Milinda Pathirage <mpathira@umail.iu.edu
> >
> wrote:
>
> > Thank you all for the ideas. I'll have a look at KafkaSystem metrics and
> > SamzaContainerMetrics.
> >
> > Milinda
> >
> > On Wed, Jun 17, 2015 at 2:38 AM, Tao Feng <fengtao04@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > One metric I could think of related to Samza job throughput is the
> > > "process-envelop" metric listed in SamzaContainerMetrics. This counter
> > > get incremented whenever the container process meaningful message(
> > >
> > >
> >
> https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/container/RunLoop.scala
> > > &&
> > >
> > >
> >
> https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala
> > > ).
> > >
> > > But this metric is more like a "QPS" type of metric .
> > >
> > > Thanks,
> > > -Tao
> > >
> > > On Tue, Jun 16, 2015 at 9:11 PM, Milinda Pathirage <
> > mpathira@umail.iu.edu>
> > > wrote:
> > >
> > > > Hi Devs,
> > > >
> > > > I was looking for a way to measure Samza job throughput and found
> that
> > > its
> > > > possible to do it via Samza's metrics reporter. But there several
> types
> > > of
> > > > metrics reported via this method. For example, TaskInstanceMetrics
> > > reports
> > > > number of messages sent. But if I wanted to get a measurement like
> > bytes
> > > > per second produced, is there a way to do that. It looks
> > > > like KafkaSystemProducerMetrics and TaskInstanceMetrics only provide
> > > number
> > > > of messages sent.
> > > >
> > > > If any of you have any experience in measuring Samza job throughput,
> > can
> > > > you please share. Really appreciate any ideas on measuring job
> > > throughput.
> > > >
> > > > Thanks
> > > > Milinda
> > > > --
> > > > Milinda Pathirage
> > > >
> > > > PhD Student | Research Assistant
> > > > School of Informatics and Computing | Data to Insight Center
> > > > Indiana University
> > > >
> > > > twitter: milindalakmal
> > > > skype: milinda.pathirage
> > > > blog: http://milinda.pathirage.org
> > > >
> > >
> >
> >
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message