hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khaled Elmeleegy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5632) Jobtracker leaves tasktrackers underutilized
Date Wed, 22 Apr 2009 04:41:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701377#action_12701377
] 

Khaled Elmeleegy commented on HADOOP-5632:
------------------------------------------

I think that the work division between the light weight and the heavy weight
heartbeats needs more thought. If running tasks status updates can be done
cheaply, we can piggyback these updates on the lightweight heartbeats. On
one extreme, each light weight heartbeat will have status updates of all the
running tasks. On the other extreme, tasks status updates are only passed
with the heavy weight heartbeats. Or somewhere in the middle, where some
light weight heartbeats can have status updates of some of the running
tasks. 

I was thinking of having the heavy weight heartbeats have a low frequency,
like every few minutes. If this would create a problem for the jobtracker as
it will not be as up-to-date with the running tasks' status as we'd need,
then running tasks status updates will need to be somehow piggybacked on the
light weight heartbeats.

I think that once we agree on what's done in each heartbeat type, we can
then decide on what to call each type of heartbeats.


Khaled





> Jobtracker leaves tasktrackers underutilized
> --------------------------------------------
>
>                 Key: HADOOP-5632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5632
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>         Environment: 2x HT 2.8GHz Intel Xeon, 3GB RAM, 4x 250GB HD linux boxes, 100 node
cluster
>            Reporter: Khaled Elmeleegy
>         Attachments: hadoop-khaled-tasktracker.10s.uncompress.timeline.pdf, hadoop-khaled-tasktracker.150ms.uncompress.timeline.pdf,
jobtracker.patch, jobtracker20.patch
>
>
> For some workloads, the jobtracker doesn't keep all the slots utilized even under heavy
load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message