storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otávio Carvalho <otav...@gmail.com>
Subject Re: Storm Performace Benchmark
Date Mon, 24 Feb 2014 18:06:57 GMT
You can also take a look at storm-perf-test (
https://github.com/yahoo/storm-perf-test/) source code.
I'm currently trying to extract some metrics, in order to develop
benchmarks for storm and other stream processors, and I thought it was
really useful.

Thanks,

Otávio.

Undergraduate Student at Federal University of Rio Grande do Sul -
http://inf.ufrgs.br
Scholarship holder at Parallel and Distributed Processing Group -
http://gppd.inf.ufrgs.br
omcarvalho@inf.ufrgs.br / @otaviocarvalho

2014-02-24 12:50 GMT-03:00 Milinda Pathirage <mpathira@umail.iu.edu>:

> Hi Padma,
>
> I think answers to your questions are there in the article you
> mentioned. Anyway I'll try to explain what needs to be done briefly.
> Note that I don't have any experience on statsd or graphite.
>
> First on sumerizing the metrics in metrics.log file. If you want to
> summerize the metrics mentioned in the article, you will have to write
> your own summarizer. It depends on what and how to summerize data you
> collected. It looks like fields in metrics.log contains information
> such as timestamp of the metrics publish event, storm host name, bolt
> identifier, metrics identifier and actual metrics value. You should be
> able to understand it by reading [2].
>
>   - It looks like people are using statsd to feed graphite[1]. And
> author of the article you mentioned also planning to use the same
> approach.
>   - In this case you need to first write a metrics consumer which
> publish metrics to statsd.
>   - Then connnect statsd and graphite according to [2].
>
> I think its possible to write a metrics consumer which directly feed
> graphite. But I am not sure whether which approach is easier.
>
> Thanks
> Milinda
>
> [1]
> http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/
> [2]
> https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/LoggingMetricsConsumer.java
>
> On Mon, Feb 24, 2014 at 8:40 AM, padma priya chitturi
> <padmapriya30@gmail.com> wrote:
> > Hi All,
> >
> > I've been using storm metrics to visualize the performance of storm (
> > http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to).
> >
> > I have included the metrics initialization code in ExclamationTopology
> code
> > and saw the metrics in metrics.log
> >
> > How can we summarize the metrics in metrics.log file. ? what do different
> > fields mean ? How can we visualize the metrics using Graphite ?
> >
> > Can someone suggest me in this..
> >
> > Thanks,
> > Padma Ch.
>
>
>
> --
> Milinda Pathirage
>
> PhD Student | Research Assistant
> School of Informatics and Computing | Data to Insight Center
> Indiana University
>
> twitter: milindalakmal
> skype: milinda.pathirage
> blog: http://milinda.pathirage.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message