incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjie Lai <mjla...@gmail.com>
Subject Re: Performance monitoring by Ganglia
Date Sat, 20 Aug 2011 00:33:44 GMT
Jon.

Regarding metrics, hbase utilizes hadoop-metrics directly:

http://hbase.apache.org/metrics.html

However I'm not sure how difficult it is to port hadoop-metrics with 
current flume reporting framework. Need to look into source code to 
understand.

This blog lists all hbase metrics:
http://blog.kenweiner.com/2010/10/monitor-hbase-hadoop-with-ganglia-on.html

-mingjie


On 08/19/2011 04:45 PM, Jonathan Hsieh wrote:
> Mingjie,
>
> I don't know much about Hadoop-metrics currently but that sounds like a
> great idea.  Do you know if other hadoop-ish (hbase?) projects use it as
> well?  If they do then I think this makes even more sense.
>
> Right now the the ganglia stuff in flume is very old and really needs to
> be re-thought.  It was essentially "borrowed" from a 0.18 version of
> hadoop I believe.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 3:19 PM, Mingjie Lai <mjlai09@gmail.com
> <mailto:mjlai09@gmail.com>> wrote:
>
>     Jon.
>
>     Have you considered to bring something like hadoop metrics framework
>     into flume? So we can decide where the metrics data goes by
>     configuration.
>
>     Otherwise we have to have another piece of code that polls and
>     parses the json and pushes to ganglia (as Torsten mentioned).
>
>     The current html and json reporting is pretty powerful. If flume can
>     push metrics to ganglia out-of-box, it would be even better
>     especially for operations.
>
>     Thanks,
>     Mingjie
>
>
>     On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>
>         I tend to agree with Torsten here about the complexity of the
>         ganglia
>         sink.  I think it easier to ask the service for metrics data.   I
>         started documenting the metrics fields reported by the json --
>         here's a
>         link:
>
>         https://cwiki.apache.org/__FLUME/monitoring-flume.html
>         <https://cwiki.apache.org/FLUME/monitoring-flume.html>
>
>         Its a work in progress -- I'll eventually document more of it in
>         the wiki.
>
>         Jon.
>
>         On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt
>         <tcurdt@vafer.org <mailto:tcurdt@vafer.org>
>         <mailto:tcurdt@vafer.org <mailto:tcurdt@vafer.org>>> wrote:
>
>             The current reporting is quite broken and does not support
>         the Ganglia
>             suggested multicast setup at all. We've got some code where
>         we fixed
>             the source and the sink but still had some trouble getting
>         it working
>             reliably (on the Ganglia sink side of things). But it didn't
>         felt
>             worth spending the time as the reporting source only polls
>         the status
>             anyway ...so we ended up having a python script polling the json
>             metrics and pushing that into Ganglia. Was easier than the
>         awkward
>             source sink setup for reporting.
>
>             cheers,
>             Torsten
>
>             On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai
>         <mjlai09@gmail.com <mailto:mjlai09@gmail.com>
>         <mailto:mjlai09@gmail.com <mailto:mjlai09@gmail.com>>> wrote:
>          > Hi.
>          >
>          > Has anyone used Ganglia to monitor flume?
>          >
>          > I'd like to have a way to monitor performance data (e.g.,
>             throughput) of our
>          > customized decorator. I saw there is Ganglia sink in the source
>             code, but
>          > don't think it's for flume itself.
>          >
>          > Can anyone point me how to use ganglia to monitor flume metrics?
>          >
>          > Thanks,
>          > Mingjie
>          >
>
>
>
>
>         --
>         // Jonathan Hsieh (shay)
>         // Software Engineer, Cloudera
>         // jon@cloudera.com <mailto:jon@cloudera.com>
>         <mailto:jon@cloudera.com <mailto:jon@cloudera.com>>
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <mailto:jon@cloudera.com>
>

Mime
View raw message