hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Iancu <daniel.ia...@1and1.ro>
Subject Re: Custom Hadoop metrics in Map Reduce jobs
Date Wed, 02 Nov 2011 09:59:07 GMT
Hi Dino
It can't be used effectively to monitor MR mappers since tasks on a node 
run as separate JVMs and each task will overwrite the same metric 
published by other tasks. You cannot aggregate values from running tasks 
before publish them because tasks don't communicate with each other.
If you publish a metric per task, this will give you tones of graphs in 
Ganglia.

On other hand I extended it the API and use it to monitor our client 
(webapp) and it works fine and provide valuable information. I have 
started a blog post to document this, will be ready soon, I'll add the 
link here if you need some examples. I wonder why this API is not a 
separate project, it can be reused in apps that don't need Hadoop. I 
could not find any java client API to publish metrics in Ganglia.

Regards
Daniel



On 11/02/2011 09:44 AM, Dino Kečo wrote:
> Hi Hadoop users,
>
> I have been reading about Hadoop metrics framework, and I was 
> wondering is it possible to create custom metrics for specific job.
>
> In my use case I want to capture some specific information per record 
> which is processed. So I was thinking to create custom code in my 
> Hadoop job to send these information using Hadoop metrics. Is this 
> possible to implement ? If so can anyone share any example how things 
> should be done?
>
> Thanks,
> dinok


Mime
View raw message