Jon.
Regarding metrics, hbase utilizes hadoop-metrics directly:
http://hbase.apache.org/metrics.html
However I'm not sure how difficult it is to port hadoop-metrics with
current flume reporting framework. Need to look into source code to
understand.
This blog lists all hbase metrics:
http://blog.kenweiner.com/2010/10/monitor-hbase-hadoop-with-ganglia-on.html
-mingjie
On 08/19/2011 04:45 PM, Jonathan Hsieh wrote:
> Mingjie,
>
> I don't know much about Hadoop-metrics currently but that sounds like a
> great idea. Do you know if other hadoop-ish (hbase?) projects use it as
> well? If they do then I think this makes even more sense.
>
> Right now the the ganglia stuff in flume is very old and really needs to
> be re-thought. It was essentially "borrowed" from a 0.18 version of
> hadoop I believe.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 3:19 PM, Mingjie Lai <mjlai09@gmail.com
> <mailto:mjlai09@gmail.com>> wrote:
>
> Jon.
>
> Have you considered to bring something like hadoop metrics framework
> into flume? So we can decide where the metrics data goes by
> configuration.
>
> Otherwise we have to have another piece of code that polls and
> parses the json and pushes to ganglia (as Torsten mentioned).
>
> The current html and json reporting is pretty powerful. If flume can
> push metrics to ganglia out-of-box, it would be even better
> especially for operations.
>
> Thanks,
> Mingjie
>
>
> On 08/19/2011 12:53 AM, Jonathan Hsieh wrote:
>
> I tend to agree with Torsten here about the complexity of the
> ganglia
> sink. I think it easier to ask the service for metrics data. I
> started documenting the metrics fields reported by the json --
> here's a
> link:
>
> https://cwiki.apache.org/__FLUME/monitoring-flume.html
> <https://cwiki.apache.org/FLUME/monitoring-flume.html>
>
> Its a work in progress -- I'll eventually document more of it in
> the wiki.
>
> Jon.
>
> On Fri, Aug 19, 2011 at 12:43 AM, Torsten Curdt
> <tcurdt@vafer.org <mailto:tcurdt@vafer.org>
> <mailto:tcurdt@vafer.org <mailto:tcurdt@vafer.org>>> wrote:
>
> The current reporting is quite broken and does not support
> the Ganglia
> suggested multicast setup at all. We've got some code where
> we fixed
> the source and the sink but still had some trouble getting
> it working
> reliably (on the Ganglia sink side of things). But it didn't
> felt
> worth spending the time as the reporting source only polls
> the status
> anyway ...so we ended up having a python script polling the json
> metrics and pushing that into Ganglia. Was easier than the
> awkward
> source sink setup for reporting.
>
> cheers,
> Torsten
>
> On Fri, Aug 19, 2011 at 1:20 AM, Mingjie Lai
> <mjlai09@gmail.com <mailto:mjlai09@gmail.com>
> <mailto:mjlai09@gmail.com <mailto:mjlai09@gmail.com>>> wrote:
> > Hi.
> >
> > Has anyone used Ganglia to monitor flume?
> >
> > I'd like to have a way to monitor performance data (e.g.,
> throughput) of our
> > customized decorator. I saw there is Ganglia sink in the source
> code, but
> > don't think it's for flume itself.
> >
> > Can anyone point me how to use ganglia to monitor flume metrics?
> >
> > Thanks,
> > Mingjie
> >
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <mailto:jon@cloudera.com>
> <mailto:jon@cloudera.com <mailto:jon@cloudera.com>>
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com <mailto:jon@cloudera.com>
>
|