hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "paul sutter (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-195) transfer map output transfer with http instead of rpc
Date Thu, 11 May 2006 15:45:06 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-195?page=comments#action_12379089 ] 

paul sutter commented on HADOOP-195:
------------------------------------


Owen,

Lovely chart, thanks for posting it. 

Are you sure that the big valley in sequential mode was the re-execution of tasks?? if not,
that looks really promising area for further measurement to see what else is going on at that
time.

It is interesting that parallel sessions dont seem to increase the transfer rate by much.
Either approach gets 50-60k files/minute, which means about 1000 files a second, which is
only about 5 files received per second per node, or 10 files per second total per node (in+out).

Does this suggest that the bottleneck is not TCP or anything in the network, since parallel
sessions should absorb these type of issues??

If so, the problem must be somewhere else, possibly disk or RPC code. Admittedly, 10 files
per second processed by a node is so feeble that it seems hard to imagine it is a spindle
problem, but  keeping an open mind:

- Are you using a single spindle for your temporary files, or are you spreading them across
all the available spindles on each node? Example:

<property>
  <name>mapred.local.dir</name>
<value>/data/tmp/hadoop/mapred/local,/data1/tmp/hadoop/mapred/local,/data2/tmp/mapred/local,/data3/tmp/mapred/local</value>
  <description>The local directory where MapReduce stores intermediate
  data files.  May be a space- or comma- separated list of
  directories on different devices in order to spread disk i/o.
  </description>
</property>

- Another way to test whether spindle contention is the issue would be to use a RAM disk for
the temporary files. Yes its a silly thing to do, but it would be a way to test the question
if you have enough RAM.


> transfer map output transfer with http instead of rpc
> -----------------------------------------------------
>
>          Key: HADOOP-195
>          URL: http://issues.apache.org/jira/browse/HADOOP-195
>      Project: Hadoop
>         Type: Improvement

>   Components: mapred
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3
>  Attachments: data-transfer-chart.pdf, netstat.log, netstat.xls
>
> The data transfer of the map output should be transfered via http instead rpc, because
rpc is very slow for this application and the timeout behavior is suboptimal. (server sends
data and client ignores it because it took more than 10 seconds to be received.)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message