hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-653) distcp can support bandwidth limiting
Date Mon, 08 Mar 2010 09:45:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842604#action_12842604
] 

Ravi Gummadi commented on MAPREDUCE-653:
----------------------------------------

Yes. But we need to make sure that a particular distcp-Job's task on a node is not hogging
the whole bandwidth available for the node(just to be fair to the other job's tasks running
on this node).
 Also at the time of launching the distcp job, we will not have enough information on the
number of tasks of this job that will run paralelly on the cluster at different points of
time(because (a) at some point few tasks of this distcp job could be running --- allowing
tasks of this distcp job to use high bandwidth per task and (b) at some other time, different
number of tasks of this distcp job running --- allowing tasks of this distcp job to use low
bandwidth per task).
So if sleeping by map task of distcp job looks bad, then distcp needs to find out a way of
identifying the number of map tasks to be run(actually, number of map tasks that can run paralelly
on the cluster at any point of time ?) so that it can set the bandwidth usage per task properly
with lower sleep time.

> distcp can support bandwidth limiting
> -------------------------------------
>
>                 Key: MAPREDUCE-653
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-653
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: distcp
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch
>
>
> distcp should support an option for user to specify the bandwidth limit for the distcp
job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message