incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: Performance monitoring by Ganglia
Date Fri, 19 Aug 2011 23:45:25 GMT
Mingjie,

I don't know much about Hadoop-metrics currently but that sounds like a
great idea.  Do you know if other hadoop-ish (hbase?) projects use it as
well?  If they do then I think this makes even more sense.

Right now the the ganglia stuff in flume is very old and really needs to be
re-thought.  It was essentially "borrowed" from a 0.18 version of hadoop I
believe.

Jon.

On Fri, Aug 19, 2011 at 3:19 PM, Mingjie Lai <mjlai09@gmail.com> wrote:

> Jon.
>
> Have you considered to bring something like hadoop metrics framework into
> flume? So we can decide where the metrics data goes by configuration.
>
> Otherwise we have to have another piece of code that polls and parses the
> json and pushes to ganglia (as Torsten mentioned).
>
> The current html and json reporting is pretty powerful. If flume can push
> metrics to ganglia out-of-box, it would be even better especially for
> operations.
>
> Thanks,
> Mingjie
>
>
> On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>
>> I tend to agree with Torsten here about the complexity of the ganglia
>> sink.  I think it easier to ask the service for metrics data.   I
>> started documenting the metrics fields reported by the json -- here's a
>> link:
>>
>> https://cwiki.apache.org/**FLUME/monitoring-flume.html<https://cwiki.apache.org/FLUME/monitoring-flume.html>
>>
>> Its a work in progress -- I'll eventually document more of it in the wiki.
>>
>> Jon.
>>
>> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt <tcurdt@vafer.org
>> <mailto:tcurdt@vafer.org>> wrote:
>>
>>    The current reporting is quite broken and does not support the Ganglia
>>    suggested multicast setup at all. We've got some code where we fixed
>>    the source and the sink but still had some trouble getting it working
>>    reliably (on the Ganglia sink side of things). But it didn't felt
>>    worth spending the time as the reporting source only polls the status
>>    anyway ...so we ended up having a python script polling the json
>>    metrics and pushing that into Ganglia. Was easier than the awkward
>>    source sink setup for reporting.
>>
>>    cheers,
>>    Torsten
>>
>>    On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai <mjlai09@gmail.com
>>    <mailto:mjlai09@gmail.com>> wrote:
>>     > Hi.
>>     >
>>     > Has anyone used Ganglia to monitor flume?
>>     >
>>     > I'd like to have a way to monitor performance data (e.g.,
>>    throughput) of our
>>     > customized decorator. I saw there is Ganglia sink in the source
>>    code, but
>>     > don't think it's for flume itself.
>>     >
>>     > Can anyone point me how to use ganglia to monitor flume metrics?
>>     >
>>     > Thanks,
>>     > Mingjie
>>     >
>>
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com <mailto:jon@cloudera.com>
>>
>>


-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

Mime
View raw message