hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5632) Jobtracker leaves tasktrackers underutilized
Date Wed, 22 Apr 2009 14:22:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701543#action_12701543
] 

Arun C Murthy commented on HADOOP-5632:
---------------------------------------

bq. ?? The sequence number seems strange.

The sequence number is necessary to ensure that lost RPCs do not lead to lost tasks... for
e.g. previously if the response to the heartbeat (containing the new tasks to be run) was
'lost', the JobTracker didn't realize it until the 'task initialization timeout' which is
a fairly long time (default of 10mins). The TaskTracker would go ahead, ignore the lost RPC,
and request tasks afresh... hence the sequence numbers.

This was the problem Owen was alluding to in https://issues.apache.org/jira/browse/HADOOP-5632?focusedCommentId=12696297&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12696297.

----

I'd like to sound a cautionary note...

The big win with HADOOP-3136 (i.e. assigning multiple tasks per heartbeat) was that it significantly
reduced the load on the JobTracker, especially on large clusters (I was running a 4000-node
cluster), since it significantly reduced the number of RPCs that the JobTracker had to handle,
especiallly coming from 4000 tasktrackers. This was felt to be very crucial as we attempted
to scale-out our clusters.

A lot of our current problems stem from the coarse-grained locking structure in the JobTracker
where we lock-up the JobTracker before we handle _any_ RPCs including heartbeats. See HADOOP-869
for more details. The contention on that single lock is significant with larger-sized clusters,
especially when running short maps... it might be good to consider HADOOP-869 a pre-requisite
for the current proposal.

Overall, I'm not against this proposal, merely stressing that we need to consider the effect
of this proposal on _much_ larger clusters before we commit to it, and we might have to do
more work than just split up the RPCs i.e. fix locking in the JobTracker. 

----

OTOH an intermediate solution might be to fix HADOOP-5129 i.e. send an heartbeat immediately
when the last running map/reduce completes, without respecting the heartbeat interval per-se.
It should get us further ahead.


> Jobtracker leaves tasktrackers underutilized
> --------------------------------------------
>
>                 Key: HADOOP-5632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5632
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>         Environment: 2x HT 2.8GHz Intel Xeon, 3GB RAM, 4x 250GB HD linux boxes, 100 node
cluster
>            Reporter: Khaled Elmeleegy
>         Attachments: hadoop-khaled-tasktracker.10s.uncompress.timeline.pdf, hadoop-khaled-tasktracker.150ms.uncompress.timeline.pdf,
jobtracker.patch, jobtracker20.patch
>
>
> For some workloads, the jobtracker doesn't keep all the slots utilized even under heavy
load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message