incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjie Lai <mjla...@gmail.com>
Subject Re: Performance monitoring by Ganglia
Date Fri, 19 Aug 2011 22:19:58 GMT
Jon.

Have you considered to bring something like hadoop metrics framework 
into flume? So we can decide where the metrics data goes by configuration.

Otherwise we have to have another piece of code that polls and parses 
the json and pushes to ganglia (as Torsten mentioned).

The current html and json reporting is pretty powerful. If flume can 
push metrics to ganglia out-of-box, it would be even better especially 
for operations.

Thanks,
Mingjie

On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
> I tend to agree with Torsten here about the complexity of the ganglia
> sink.  I think it easier to ask the service for metrics data.   I
> started documenting the metrics fields reported by the json -- here's a
> link:
>
> https://cwiki.apache.org/FLUME/monitoring-flume.html
>
> Its a work in progress -- I'll eventually document more of it in the wiki.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tcurdt@vafer.org
> <mailto:tcurdt@vafer.org>> wrote:
>
>     The current reporting is quite broken and does not support the Ganglia
>     suggested multicast setup at all. We've got some code where we fixed
>     the source and the sink but still had some trouble getting it working
>     reliably (on the Ganglia sink side of things). But it didn't felt
>     worth spending the time as the reporting source only polls the status
>     anyway ...so we ended up having a python script polling the json
>     metrics and pushing that into Ganglia. Was easier than the awkward
>     source sink setup for reporting.
>
>     cheers,
>     Torsten
>
>     On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mjlai09@gmail.com
>     <mailto:mjlai09@gmail.com>> wrote:
>      > Hi.
>      >
>      > Has anyone used Ganglia to monitor flume?
>      >
>      > I'd like to have a way to monitor performance data (e.g.,
>     throughput) of our
>      > customized decorator. I saw there is Ganglia sink in the source
>     code, but
>      > don't think it's for flume itself.
>      >
>      > Can anyone point me how to use ganglia to monitor flume metrics?
>      >
>      > Thanks,
>      > Mingjie
>      >
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <mailto:jon@cloudera.com>
>

Mime
View raw message