hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khaled Elmeleegy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5632) Jobtracker leaves tasktrackers underutilized
Date Sun, 07 Jun 2009 20:10:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717091#action_12717091
] 

Khaled Elmeleegy commented on HADOOP-5632:
------------------------------------------

For a similar workload (20s maps), slot utilization was ~60% for the
unpatched JT. Now it's 99.8%.

There are two questions here:

- Does the patch keep all the slots busy? Previous experiments have shown
that it does.
- Can the patched JT keep up with the load from a large cluster, as the
patched JT could potentially be doing more work than the unpatched one?

These new experiments was trying to answer the later question. Results
suggest that the patched JT can easily handle large clusters.





> Jobtracker leaves tasktrackers underutilized
> --------------------------------------------
>
>                 Key: HADOOP-5632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5632
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>         Environment: 2x HT 2.8GHz Intel Xeon, 3GB RAM, 4x 250GB HD linux boxes, 100 node
cluster
>            Reporter: Khaled Elmeleegy
>         Attachments: hadoop-khaled-tasktracker.10s.uncompress.timeline.pdf, hadoop-khaled-tasktracker.150ms.uncompress.timeline.pdf,
jobtracker.patch, jobtracker20.patch
>
>
> For some workloads, the jobtracker doesn't keep all the slots utilized even under heavy
load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message