hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From omal...@apache.org
Subject svn commit: r644900 - in /hadoop/core/trunk: CHANGES.txt src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java
Date Fri, 04 Apr 2008 20:40:09 GMT
Author: omalley
Date: Fri Apr  4 13:40:08 2008
New Revision: 644900

URL: http://svn.apache.org/viewvc?rev=644900&view=rev
Log:
HADOOP-3168. Reduce the amount of logging in streaming to an
exponentially increasing number of records (up to 10,000
records/log). Contributed by Zheng Shao.

Modified:
    hadoop/core/trunk/CHANGES.txt
    hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java

Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=644900&r1=644899&r2=644900&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Fri Apr  4 13:40:08 2008
@@ -249,6 +249,10 @@
     HADOOP-3140. Doesn't add a task in the commit queue if the task hadn't
     generated any output. (Amar Kamat via ddas)
 
+    HADOOP-3168. Reduce the amount of logging in streaming to an
+    exponentially increasing number of records (up to 10,000
+    records/log). (Zheng Shao via omalley)
+ 
   BUG FIXES
 
     HADOOP-2195. '-mkdir' behaviour is now closer to Linux shell in case of

Modified: hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java?rev=644900&r1=644899&r2=644900&view=diff
==============================================================================
--- hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java
(original)
+++ hadoop/core/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java
Fri Apr  4 13:40:08 2008
@@ -511,8 +511,11 @@
       String info = numRecInfo();
       logprintln(info);
       logflush();
-      //nextRecReadLog_ *= 10;
-      nextRecReadLog_ += 100;
+      if (nextRecReadLog_ < 100000) {
+	  nextRecReadLog_ *= 10;
+      } else {
+	  nextRecReadLog_ += 100000;
+      }
     }
   }
 



Mime
View raw message