hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-195) transfer map output transfer with http instead of rpc
Date Fri, 12 May 2006 06:32:09 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-195?page=comments#action_12379174 ] 

eric baldeschwieler commented on HADOOP-195:
--------------------------------------------

The network core has a lot more bandwidth than that.  Each rack has a 1Gbit uplink, although
this is a crufty enough network (oldest gear we have) that we may find some odd outliers.
 But it can be assumed that we have at least 30mBits of bandwidth simultaneously from ever
node to every other.  And the write tests are behaving as if we do.  So we can focus on finding
other explainations.  We're also starting benchmarking on a newer set of nodes with ~100mBits
simultaneously all to all.

> transfer map output transfer with http instead of rpc
> -----------------------------------------------------
>
>          Key: HADOOP-195
>          URL: http://issues.apache.org/jira/browse/HADOOP-195
>      Project: Hadoop
>         Type: Improvement

>   Components: mapred
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3
>  Attachments: MapFileSimulator.java, data-transfer-chart.pdf, mapfilesimulator-big.txt,
mapfilesimulator-sort2.txt, netstat.log, netstat.xls
>
> The data transfer of the map output should be transfered via http instead rpc, because
rpc is very slow for this application and the timeout behavior is suboptimal. (server sends
data and client ignores it because it took more than 10 seconds to be received.)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message