hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1338) Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers of files
Date Tue, 08 May 2007 17:03:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494344
] 

Doug Cutting commented on HADOOP-1338:
--------------------------------------

As a performance-enhancement, any patch for this must demonstrate a significant performance
improvement before it can be committed.  I suspect that simply caching a fixed number of connections
will not provide a significant performance enhancement.  Rather, we might attempt, when contacting
a node, to transfer all available output that resides on that node.  So, instead of randomly
shuffling the map output locations, a reduce task might first group locations by host, then
randomly shuffle, fetching batches from each host.  However, if the shuffle is already keeping
up with the map, even this may not improve things much, since each map node may tend to only
have a single new output at a time.

> Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers
of files
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1338
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1338
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>             Fix For: 0.14.0
>
>
> We should do transfers of map outputs at the granularity of  *total-bytes-transferred*
rather than the current way of transferring a single file and then closing the connection
to the server. A single TaskTracker might have a couple of map output files for a given reduce,
and we should transfer multiple of them (upto a certain total size) in a single connection
to the TaskTracker. Using HTTP-1.1's keep-alive connection would help since it would keep
the connection open for more than one file transfer. We should limit the transfers to a certain
size so that we don't hold up a jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message