hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5931) Collect information about number of tasks succeeded / total per time unit for a tasktracker.
Date Thu, 11 Jun 2009 00:53:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718268#action_12718268

Sharad Agarwal commented on HADOOP-5931:

bq. I am assuming the moving window mechanism would be flexible enough to add new bucket sizes
as required. 
Yes. I am planning to use and extend metric framework available in core, thru which custom
window/bucket sizes can be defined.

bq. Regarding having the computation on the tasktracker, and reporting the status via status,
one problem is that if we want to change the bucket size, it would involve a change in the
status object.
To avoid that, instead of above fields, we can have say List<MetricInfo> metrics field
in TaskTrackerStatus where MetricInfo could be:
class MetricInfo {
String name;
int tasks;
int tasksSucceeded;
Here name would be the name of the metrics. e.q. "lasthour", "lastday" etc. which could be
configured in the metrics property file. 

bq. Also, one requirement for this is to store this information on the JobTracker. Can you
describe how this will be stored, mechanics with respect to lost tasktrackers etc ?
Currently jobtracker doesn't store any information about lost tasktrackers. Storing info about
lost trackers is not trivial and demands a separate jira issue. Consider the case of tracker
getting lost and never coming back or coming back at different port. The jobtracker data structures
need to be cleaned up for such trackers otherwise those data structures would be lying forever.

bq. Will this information be available if the JobTracker restarts ?
Yes. Since this info is propagated from Tasktracker, it would be available after jobtracker

> Collect information about number of tasks succeeded / total per time unit for a tasktracker.

> ---------------------------------------------------------------------------------------------
>                 Key: HADOOP-5931
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5931
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
> Collecting information of number of tasks succeeded / total per tasktracker and being
able to see these counts per hour, day and since start time will help reason about things
like the blacklisting strategy.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message