incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Hauser <ewhau...@gmail.com>
Subject Re: Performance monitoring by Ganglia
Date Fri, 19 Aug 2011 22:42:09 GMT
FWIW, there is a Ruby script in the ostrich repository for parsing
JSON stats and sending them to Ganglia that could be slightly modified
to work with Flume:

https://github.com/twitter/ostrich/blob/master/src/scripts/json_stats_fetcher.rb


On Fri, Aug 19, 2011 at 6:19 PM, Mingjie Lai <mjlai09@gmail.com> wrote:
> Jon.
>
> Have you considered to bring something like hadoop metrics framework into
> flume? So we can decide where the metrics data goes by configuration.
>
> Otherwise we have to have another piece of code that polls and parses the
> json and pushes to ganglia (as Torsten mentioned).
>
> The current html and json reporting is pretty powerful. If flume can push
> metrics to ganglia out-of-box, it would be even better especially for
> operations.
>
> Thanks,
> Mingjie
>
> On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>>
>> I tend to agree with Torsten here about the complexity of the ganglia
>> sink.  I think it easier to ask the service for metrics data.   I
>> started documenting the metrics fields reported by the json -- here's a
>> link:
>>
>> https://cwiki.apache.org/FLUME/monitoring-flume.html
>>
>> Its a work in progress -- I'll eventually document more of it in the wiki.
>>
>> Jon.
>>
>> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tcurdt@vafer.org
>> <mailto:tcurdt@vafer.org>> wrote:
>>
>>    The current reporting is quite broken and does not support the Ganglia
>>    suggested multicast setup at all. We've got some code where we fixed
>>    the source and the sink but still had some trouble getting it working
>>    reliably (on the Ganglia sink side of things). But it didn't felt
>>    worth spending the time as the reporting source only polls the status
>>    anyway ...so we ended up having a python script polling the json
>>    metrics and pushing that into Ganglia. Was easier than the awkward
>>    source sink setup for reporting.
>>
>>    cheers,
>>    Torsten
>>
>>    On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mjlai09@gmail.com
>>    <mailto:mjlai09@gmail.com>> wrote:
>>     > Hi.
>>     >
>>     > Has anyone used Ganglia to monitor flume?
>>     >
>>     > I'd like to have a way to monitor performance data (e.g.,
>>    throughput) of our
>>     > customized decorator. I saw there is Ganglia sink in the source
>>    code, but
>>     > don't think it's for flume itself.
>>     >
>>     > Can anyone point me how to use ganglia to monitor flume metrics?
>>     >
>>     > Thanks,
>>     > Mingjie
>>     >
>>
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com <mailto:jon@cloudera.com>
>>
>

Mime
View raw message