hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1338) Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers of files
Date Thu, 18 Dec 2008 14:57:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657793#action_12657793

Jothi Padmanabhan commented on HADOOP-1338:

To see if actually pulling more data at one short is beneficial or not, we tried the following
experiment. We ran the loadgen example with 50K maps, 2MB per map. We ran with num reduces
= 1, 2 and 4.

The idea here is that with increasing number of reducers, each reducer would pull in lesser
amount of intermediate data to fetch from a given map.

We found the shuffie times as
Num Reducers = 1, 5800s
Num Reducers = 2, 3120s
Num Reducers = 4, 1660s 

In essence, when we reduce the amount fetched by a factor of half, the time actually did not
fall by half, but was slightly more than half.  Do these results actually imply that batching
of maps could be beneficial? Thoughts?

> Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers
of files
> --------------------------------------------------------------------------------------------------
>                 Key: HADOOP-1338
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1338
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
> We should do transfers of map outputs at the granularity of  *total-bytes-transferred*
rather than the current way of transferring a single file and then closing the connection
to the server. A single TaskTracker might have a couple of map output files for a given reduce,
and we should transfer multiple of them (upto a certain total size) in a single connection
to the TaskTracker. Using HTTP-1.1's keep-alive connection would help since it would keep
the connection open for more than one file transfer. We should limit the transfers to a certain
size so that we don't hold up a jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message