hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1338) Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers of files
Date Wed, 19 Nov 2008 00:41:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648841#action_12648841
] 

Owen O'Malley commented on HADOOP-1338:
---------------------------------------

What is the advantage of sending multiple maps ids at once rather than using keep alive? It
is much better to just use the standard http keep alive paradigm. This should be put on hold
until we have upgraded to Jetty 6. I suspect that part of the problem may be the use of Java's
url getter. We should compare the performance of Java URL versus Apache HttpClient.

> Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers
of files
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1338
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1338
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>
> We should do transfers of map outputs at the granularity of  *total-bytes-transferred*
rather than the current way of transferring a single file and then closing the connection
to the server. A single TaskTracker might have a couple of map output files for a given reduce,
and we should transfer multiple of them (upto a certain total size) in a single connection
to the TaskTracker. Using HTTP-1.1's keep-alive connection would help since it would keep
the connection open for more than one file transfer. We should limit the transfers to a certain
size so that we don't hold up a jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message