Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Message-ID: <16826792.172841288616246024.JavaMail.jira@thor>
Date: Mon, 1 Nov 2010 08:57:26 -0400 (EDT)
From: "Liyin Liang (JIRA)" <jira@apache.org>
To: mapreduce-dev@hadoop.apache.org
Subject: [jira] Created: (MAPREDUCE-2168) We should  implement limits on
 shuffle connections to TaskTracker per job
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

We should  implement limits on shuffle connections to TaskTracker per job
-------------------------------------------------------------------------

                 Key: MAPREDUCE-2168
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2168
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
            Reporter: Liyin Liang


As trailing map tasks will be attacked by all reduces simultaneously, all the worker threads that for the http server of a TaskTracker may be occupied  by one job's reduce tasks to fetch map outputs. Then this tasktracker's iowait and load will be very high (100+ in our cluster, we set tasktracker.http.threads with 100). What's more, other job's reduces have to wait some time (may be several minutes) to connect to the TaskTracker to fetch there map's outputs.
So I think we should implement limits on shuffle connections:
1. limit the worker threads' number maybe percent  occupied  the same job's reduces ;
2. limit the worker threads' number serving the same map output simultaneously.
Thoughts? 

ps: we are using hadoop 0.19.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.