hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5931) Collect information about number of tasks succeeded / total per time unit for a tasktracker.
Date Tue, 16 Jun 2009 12:31:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720094#action_12720094
] 

Sharad Agarwal commented on HADOOP-5931:
----------------------------------------

Had an off line discussion with Devaraj/Eric, the concern raised is that metric context is
an export interface and instead of using it, we should collect the metrics natively in hadoop.
Administrators should not be able to remove this metric as it may in future used by Jobtracker
to make decisions. Right?
Let me clarify a bit. Please note that only time windows are configured in the metric properties,
and not the actual metric name which gets collected. Also a new context name is defined "tasktracker"
(Refer hadoop-metrics.properties in patch) . So it does not come in between the existing metric
contexts. Those can continue to be chukwa/ganglia etc.
If this doesn't sound like a good idea, I see few options:
1. Give a better name to the added context say "core-mapred", so that administrators don't
override it. It would serve only to add/remove time windows.

2. Do not use Metrics api. Expose the time window configuration via mapred-site.xml.

3. Don't expose the configuration at all and have fixed windows, say "last hour" and "last
day".

I went with extending the metrics API because I thought that it would help to collect any
other existing metrics in time windows without making much change to the code. For example
if we want to collect "mapred" metrics in time windows, then "mapred" context can point to
the Composite context, which can be configured to use multiple contexts, one being time window
context.

Thoughts?

> Collect information about number of tasks succeeded / total per time unit for a tasktracker.

> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5931
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5931
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 5931_v1.patch, 5931_v2.patch
>
>
> Collecting information of number of tasks succeeded / total per tasktracker and being
able to see these counts per hour, day and since start time will help reason about things
like the blacklisting strategy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message