hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5632) Jobtracker leaves tasktrackers underutilized
Date Wed, 22 Apr 2009 14:28:48 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701550#action_12701550
] 

Arun C Murthy commented on HADOOP-5632:
---------------------------------------

bq. If running tasks status updates can be done cheaply, we can piggyback these updates on
the lightweight heartbeats.

Uh, that pretty much all we do on the current heartbeat, along with getTasksToKill, no? We
should consider fixing getTasksToKill, after which we are back at the current heartbeat.

bq. I was thinking of having the heavy weight heartbeats have a low frequency, like every
few minutes.

-1
That's too large, especially for jobs where latency is important. The 'heavy' heartbeats cannot
be much slower that what we have currently.

> Jobtracker leaves tasktrackers underutilized
> --------------------------------------------
>
>                 Key: HADOOP-5632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5632
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>         Environment: 2x HT 2.8GHz Intel Xeon, 3GB RAM, 4x 250GB HD linux boxes, 100 node
cluster
>            Reporter: Khaled Elmeleegy
>         Attachments: hadoop-khaled-tasktracker.10s.uncompress.timeline.pdf, hadoop-khaled-tasktracker.150ms.uncompress.timeline.pdf,
jobtracker.patch, jobtracker20.patch
>
>
> For some workloads, the jobtracker doesn't keep all the slots utilized even under heavy
load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message