hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sur...@apache.org
Subject svn commit: r1363885 - in /hadoop/common/branches/branch-1: CHANGES.txt src/mapred/org/apache/hadoop/mapred/TaskTracker.java
Date Fri, 20 Jul 2012 18:08:53 GMT
Author: suresh
Date: Fri Jul 20 18:08:53 2012
New Revision: 1363885

URL: http://svn.apache.org/viewvc?rev=1363885&view=rev
Log:
MAPREDUCE-4399. Change the Jetty response buffer size to improve shuffle performance. Contributed
by Luke Lu.

Modified:
    hadoop/common/branches/branch-1/CHANGES.txt
    hadoop/common/branches/branch-1/src/mapred/org/apache/hadoop/mapred/TaskTracker.java

Modified: hadoop/common/branches/branch-1/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1363885&r1=1363884&r2=1363885&view=diff
==============================================================================
--- hadoop/common/branches/branch-1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1/CHANGES.txt Fri Jul 20 18:08:53 2012
@@ -222,6 +222,9 @@ Release 1.1.0 - unreleased
     HDFS-2465. Add HDFS support for fadvise readahead and drop-behind.
     (todd, backported by Brandon Li via suresh)
 
+    MAPREDUCE-4399. Change the Jetty response buffer size to improve 
+    shuffle performance. (Luke Lu via suresh)
+
   BUG FIXES
 
     MAPREDUCE-4087. [Gridmix] GenerateDistCacheData job of Gridmix can

Modified: hadoop/common/branches/branch-1/src/mapred/org/apache/hadoop/mapred/TaskTracker.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/mapred/org/apache/hadoop/mapred/TaskTracker.java?rev=1363885&r1=1363884&r2=1363885&view=diff
==============================================================================
--- hadoop/common/branches/branch-1/src/mapred/org/apache/hadoop/mapred/TaskTracker.java (original)
+++ hadoop/common/branches/branch-1/src/mapred/org/apache/hadoop/mapred/TaskTracker.java Fri
Jul 20 18:08:53 2012
@@ -3867,7 +3867,8 @@ public class TaskTracker implements MRCo
   public static class MapOutputServlet extends HttpServlet {
     private static final long serialVersionUID = 1L;
     private static final int MAX_BYTES_TO_READ = 64 * 1024;
-    
+    // work around jetty internal buffering issues
+    private static final int RESPONSE_BUFFER_SIZE = MAX_BYTES_TO_READ + 16;
     private static LRUCache<String, Path> fileCache = new LRUCache<String, Path>(FILE_CACHE_SIZE);
     private static LRUCache<String, Path> fileIndexCache = new LRUCache<String,
Path>(FILE_CACHE_SIZE);
     
@@ -3969,10 +3970,9 @@ public class TaskTracker implements MRCo
         //set the custom "for-reduce-task" http header to the reduce task number
         //for which this map output is being transferred
         response.setHeader(FOR_REDUCE_TASK, Integer.toString(reduce));
-        
-        //use the same buffersize as used for reading the data from disk
-        response.setBufferSize(MAX_BYTES_TO_READ);
-        
+
+        response.setBufferSize(RESPONSE_BUFFER_SIZE);
+
         /**
          * Read the data from the sigle map-output file and
          * send it to the reducer.



Mime
View raw message