Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <856720453.1233660359774.JavaMail.jira@brutus>
Date: Tue, 3 Feb 2009 03:25:59 -0800 (PST)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-1338) Improve the shuffle phase by using
 the "connection: keep-alive" and doing batch transfers of files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jothi Padmanabhan updated HADOOP-1338:
--------------------------------------

    Attachment: hadoop-1338-v1.patch

Attached a patch that batches fetch of map outputs.
The maximum size that needs to be pulled from a tasktracker per connection is configurable.
The number of maps requested is adapted. For the first 1%, 1 map is requested per connection. Subsequently, upon learning the average size of maps, the number of maps to request is determined using the maximum size. The number of maps is still bounded by a limit imposed on the size of the HTTP request that the servlet can handle.

> Improve the shuffle phase by using the "connection: keep-alive" and doing batch transfers of files
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-1338
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1338
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Jothi Padmanabhan
>         Attachments: hadoop-1338-v1.patch
>
>
> We should do transfers of map outputs at the granularity of  *total-bytes-transferred* rather than the current way of transferring a single file and then closing the connection to the server. A single TaskTracker might have a couple of map output files for a given reduce, and we should transfer multiple of them (upto a certain total size) in a single connection to the TaskTracker. Using HTTP-1.1's keep-alive connection would help since it would keep the connection open for more than one file transfer. We should limit the transfers to a certain size so that we don't hold up a jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.