hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jo...@apache.org
Subject svn commit: r750552 - in /hadoop/core/trunk: CHANGES.txt src/docs/src/documentation/content/xdocs/mapred_tutorial.xml src/docs/src/documentation/content/xdocs/site.xml src/docs/src/documentation/content/xdocs/streaming.xml
Date Thu, 05 Mar 2009 19:07:38 GMT
Author: johan
Date: Thu Mar  5 19:07:37 2009
New Revision: 750552

URL: http://svn.apache.org/viewvc?rev=750552&view=rev
Log:
HADOOP-5317. Provide documentation for LazyOutput Feature. (Jothi Padmanabhan via johan)

Modified:
    hadoop/core/trunk/CHANGES.txt
    hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
    hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml
    hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml

Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Thu Mar  5 19:07:37 2009
@@ -146,6 +146,9 @@
     HADOOP-4546. Fix DF reporting for AIX. (Bill Habermaas via cdouglas)
 
     HADOOP-5023. Add Tomcat support to HdfsProxy. (Zhiyong Zhang via cdouglas)
+    
+    HADOOP-5317. Provide documentation for LazyOutput Feature. 
+    (Jothi Padmanabhan via johan)
 
   OPTIMIZATIONS
 

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Thu Mar
 5 19:07:37 2009
@@ -1579,6 +1579,25 @@
         <code>OutputFormat</code>.</p>
 
         <section>
+        <title>Lazy Output Creation</title>
+        <p>It is possible to delay creation of output until the first write attempt

+           by using <a href="ext:api/org/apache/hadoop/mapred/lib/lazyoutputformat">
+           LazyOutputFormat</a>. This is particularly useful in preventing the 
+           creation of zero byte files when there is no call to output.collect 
+           (or Context.write). This is achieved by calling the static method 
+           <code>setOutputFormatClass</code> of <code>LazyOutputFormat</code>

+           with the intended <code>OutputFormat</code> as the argument. The following
example 
+           shows how to delay creation of files when using the <code>TextOutputFormat</code>
+        </p>
+
+        <p>
+        <code> import org.apache.hadoop.mapred.lib.LazyOutputFormat;</code> <br/>
+        <code> LazyOutputFormat.setOutputFormatClass(conf, TextOutputFormat.class);</code>
+        </p>
+         
+        </section>
+
+        <section>
         <title>OutputCommitter</title>
         
         <p><a href="ext:api/org/apache/hadoop/mapred/outputcommitter">

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/site.xml Thu Mar  5 19:07:37
2009
@@ -236,6 +236,7 @@
                 <hashpartitioner href="HashPartitioner.html" />
                 <keyfieldbasedpartitioner href="KeyFieldBasedPartitioner.html" />
                 <keyfieldbasedcomparator href="KeyFieldBasedComparator.html" />
+                <lazyoutputformat href="LazyOutputFormat.html" />
                 <aggregate href="aggregate/">
                   <package-summary href="package-summary.html" />
                 </aggregate>

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml?rev=750552&r1=750551&r2=750552&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/streaming.xml Thu Mar  5 19:07:37
2009
@@ -239,6 +239,7 @@
 
 <tr><td> -inputreader JavaClassName </td><td> Optional </td><td>
For backwards-compatibility: specifies a record reader class (instead of an input format class)
</td></tr>
 <tr><td> -verbose </td><td> Optional </td><td> Verbose
output </td></tr>
+<tr><td> -lazyOutput </td><td> Optional </td><td> Create
output lazily. For example, if the output format is based on FileOutputFormat, the output
file is created only on the first call to output.collect (or Context.write)</td></tr>
 </table>
 <p>
 Streaming support Hadoop generic command line options. 



Mime
View raw message