incubator-hcatalog-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From khorg...@apache.org
Subject svn commit: r1381506 - in /incubator/hcatalog/trunk: CHANGES.txt src/docs/src/documentation/content/xdocs/inputoutput.xml src/docs/src/documentation/content/xdocs/loadstore.xml
Date Thu, 06 Sep 2012 08:09:05 GMT
Author: khorgath
Date: Thu Sep  6 08:09:04 2012
New Revision: 1381506

URL: http://svn.apache.org/viewvc?rev=1381506&view=rev
Log:
HCATALOG-431 document hcat type to java class/pig type mapping (lefty via khorgath)

Modified:
    incubator/hcatalog/trunk/CHANGES.txt
    incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml
    incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml

Modified: incubator/hcatalog/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/incubator/hcatalog/trunk/CHANGES.txt?rev=1381506&r1=1381505&r2=1381506&view=diff
==============================================================================
--- incubator/hcatalog/trunk/CHANGES.txt (original)
+++ incubator/hcatalog/trunk/CHANGES.txt Thu Sep  6 08:09:04 2012
@@ -38,6 +38,8 @@ Trunk (unreleased changes)
   HCAT-427 Document storage-based authorization (lefty via gates)
 
   IMPROVEMENTS
+  HCAT-431 document hcat type to java class/pig type mapping (lefty via khorgath)
+
   HCAT-492 Document CTAS workaround for Hive with JSON serde (lefty via khorgath)
 
   HCAT-487 HCatalog should tolerate a user-defined amount of bad records (traviscrawford)

Modified: incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml
URL: http://svn.apache.org/viewvc/incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml?rev=1381506&r1=1381505&r2=1381506&view=diff
==============================================================================
--- incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml (original)
+++ incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml Thu
Sep  6 08:09:04 2012
@@ -149,6 +149,84 @@ will be returned.</p>
 
 </section>
 
+<!-- ==================================================================== --> 
+<section>
+    <title>HCatRecord</title>
+
+<p>HCatRecord is the type supported for storing values in HCatalog tables.</p>
+<p>The types in an HCatalog table schema determine the types of objects returned for
different fields in HCatRecord. This table shows the mappings between Java classes for MapReduce
programs and HCatalog data types:</p>
+
+<table>
+  <tr>
+    <th><p class="center">HCatalog Data Type</p></th>
+    <th><p class="center">Java Class in MapReduce</p></th>
+    <th><p class="cell">Values</p></th>
+  </tr>
+  <tr>
+    <td><p class="center">TINYINT</p></td>
+    <td><p class="center">java.lang.Byte</p></td>
+    <td><p class="cell">-128 to 127</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">SMALLINT</p></td>
+    <td><p class="center">java.lang.Short</p></td>
+    <td><p class="cell">-2<sup>15</sup> to 2<sup>15</sup>-1
(-32,768 to 32,767)</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">INT</p></td>
+    <td><p class="center">java.lang.Integer</p></td>
+    <td><p class="cell">-2<sup>31</sup> to 2<sup>31</sup>-1
(-2,147,483,648 to 2,147,483,647)</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">BIGINT</p></td>
+    <td><p class="center">java.lang.Long</p></td>
+    <td><p class="cell">-2<sup>63</sup> to 2<sup>63</sup>-1
(-9,223,372,036,854,775,808 to      9,223,372,036,854,775,807)</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">BOOLEAN</p></td>
+    <td><p class="center">java.lang.Boolean</p></td>
+    <td><p class="cell">true or false</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">FLOAT</p></td>
+    <td><p class="center">java.lang.Float</p></td>
+    <td><p class="cell">single-precision floating-point value</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">DOUBLE</p></td>
+    <td><p class="center">java.lang.Double</p></td>
+    <td><p class="cell">double-precision floating-point value</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">BINARY</p></td>
+    <td><p class="center">byte[]</p></td>
+    <td><p class="cell">binary data</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">STRING</p></td>
+    <td><p class="center">java.lang.String</p></td>
+    <td><p class="cell">character string</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">STRUCT</p></td>
+    <td><p class="center">java.util.List</p></td>
+    <td><p class="cell">structured data</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">ARRAY</p></td>
+    <td><p class="center">java.util.List</p></td>
+    <td><p class="cell">values of one data type</p></td>
+  </tr>
+  <tr>
+    <td><p class="center">MAP</p></td>
+    <td><p class="center">java.util.Map</p></td>
+    <td><p class="cell">key-value pairs</p></td>
+  </tr>
+</table>
+
+</section>
+
+<!-- ==================================================================== --> 
 <section>
     <title>Running MapReduce with HCatalog</title>
 <p>

Modified: incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml
URL: http://svn.apache.org/viewvc/incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml?rev=1381506&r1=1381505&r2=1381506&view=diff
==============================================================================
--- incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml (original)
+++ incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml Thu Sep
 6 08:09:04 2012
@@ -68,51 +68,131 @@ immediately following the load statement
 <section> 
   <title>HCatalog Data Types</title>
 
-<p>Restrictions apply to the types of columns HCatLoader can read.</p>
-<p>HCatLoader  can read <strong>only</strong> the data types listed in
the table. 
+<p>Restrictions apply to the types of columns HCatLoader can read from HCatalog-managed
tables.</p>
+<p>HCatLoader can read <em><strong>only</strong></em> the data
types listed in the table below.
 The table shows how Pig will interpret the HCatalog data type.</p>
-   <table>
-        <tr>
-            <td>
-               <p><strong>HCatalog Data Type</strong></p>
-            </td>
-            <td>
-               <p><strong>Pig Data Type</strong></p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>primitives (int, long, float, double, string) </p>
-            </td>
-            <td>
-               <p>int, long, float, double, string to chararray </p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>map (key type should be string, valuetype must be string)</p>
-            </td>
-            <td>
-               <p>map </p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>List&lt;any type&gt; </p>
-            </td>
-            <td>
-               <p>bag </p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>struct&lt;any type fields&gt; </p>
-            </td>
-            <td>
-               <p>tuple </p>
-            </td>
+  <table>
+    <tr>
+        <th>
+           <p class="center">Primitives</p>
+        </th>
+        <th>
+           <p class="center"> </p>
+        </th>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell"><strong>HCatalog Data Type</strong></p>
+        </td>
+        <td>
+           <p class="cell"><strong>Pig Data Type</strong></p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">int</p>
+        </td>
+        <td>
+           <p class="cell">int</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">long</p>
+        </td>
+        <td>
+           <p class="cell">long</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">float</p>
+        </td>
+        <td>
+           <p class="cell">float</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">double</p>
+        </td>
+        <td>
+           <p class="cell">double</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">string</p>
+        </td>
+        <td>
+           <p class="cell">chararray</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">boolean</p>
+        </td>
+        <td>
+           <p class="cell">boolean</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">binary</p>
+        </td>
+        <td>
+           <p class="cell">bytearray</p>
+        </td>
+    </tr>
+</table>
+<table>
+    <tr>
+        <th>
+           <p class="center">Complex Types</p>
+        </th>
+        <th>
+           <p class="center"></p>
+        </th>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell"><strong>HCatalog Data Type</strong></p>
+        </td>
+        <td>
+           <p class="cell"><strong>Pig Data Type</strong></p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">map<br/>(key type should be string)</p>
+        </td>
+        <td>
+           <p class="cell">map </p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">List<em>&lt;any type&gt;</em> </p>
+        </td>
+        <td>
+           <p class="cell">bag </p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">struct<em>&lt;any type fields&gt;</em>
</p>
+        </td>
+        <td>
+           <p class="cell">tuple </p>
+        </td>
     </tr>
  </table>
+
+<p><br/></p>
+<p>Currently HCatLoader cannot map the smallint and tinyint datatypes to Pig data types.
+This issue exists in HCatalog version 0.4.0;
+the fix for Jira issue <a href="https://issues.apache.org/jira/browse/HCATALOG-425">HCATALOG-425</a>
should be available in HCatalog version 0.5.0.</p>
+
 </section> 
 
  <!-- ==================================================================== --> 
@@ -355,51 +435,132 @@ HCatStorer with no argument:</p>
 <section>
   <title>HCatalog Data Types</title>
 
-<p>Restrictions apply to the types of columns HCatStorer can write.</p>
-<p>HCatStorer can write <strong>only</strong> the data types listed in
the table. 
+<p>Restrictions apply to the types of columns HCatStorer can write to HCatalog-managed
tables.</p>
+<p>HCatStorer can write <em><strong>only</strong></em> the
data types listed in the table.
 The table shows how Pig will interpret the HCatalog data type.</p>
-   <table>
-        <tr>
-            <td>
-               <p><strong>HCatalog Data Type</strong></p>
-            </td>
-            <td>
-               <p><strong>Pig Data Type</strong></p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>primitives (int, long, float, double, string) </p>
-            </td>
-            <td>
-               <p>int, long, float, double, string to chararray </p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>map (key type should be string, valuetype must be string)</p>
-            </td>
-            <td>
-               <p>map </p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>List&lt;any type&gt; </p>
-            </td>
-            <td>
-               <p>bag </p>
-            </td>
-    </tr>
-    <tr>
-            <td>
-               <p>struct&lt;any type fields&gt; </p>
-            </td>
-            <td>
-               <p>tuple </p>
-            </td>
+
+  <table>
+    <tr>
+        <th>
+           <p class="center">Primitives</p>
+        </th>
+        <th>
+           <p class="center"> </p>
+        </th>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell"><strong>Pig Data Type</strong></p>
+        </td>
+        <td>
+           <p class="cell"><strong>HCatalog Data Type</strong></p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">int</p>
+        </td>
+        <td>
+           <p class="cell">int</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">long</p>
+        </td>
+        <td>
+           <p class="cell">long</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">float</p>
+        </td>
+        <td>
+           <p class="cell">float</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">double</p>
+        </td>
+        <td>
+           <p class="cell">double</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">chararray</p>
+        </td>
+        <td>
+           <p class="cell">string</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">boolean</p>
+        </td>
+        <td>
+           <p class="cell">boolean</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">bytearray</p>
+        </td>
+        <td>
+           <p class="cell">binary</p>
+        </td>
+    </tr>
+</table>
+<table>
+    <tr>
+        <th>
+           <p class="center">Complex Types</p>
+        </th>
+        <th>
+           <p class="center"> </p>
+        </th>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell"><strong>Pig Data Type</strong></p>
+        </td>
+        <td>
+           <p class="cell"><strong>HCatalog Data Type</strong></p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">map</p>
+        </td>
+        <td>
+           <p class="cell">map<br/>(key type should be string)</p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">bag </p>
+        </td>
+        <td>
+           <p class="cell">List<em>&lt;any type&gt;</em> </p>
+        </td>
+    </tr>
+    <tr>
+        <td>
+           <p class="cell">tuple </p>
+        </td>
+        <td>
+           <p class="cell">struct<em>&lt;any type fields&gt;</em>
</p>
+        </td>
     </tr>
  </table>
+
+<p><br/></p>
+<p>Currently HCatLoader cannot map the smallint and tinyint datatypes to Pig data types.
+This issue exists in HCatalog version 0.4.0;
+the fix for Jira issue <a href="https://issues.apache.org/jira/browse/HCATALOG-425">HCATALOG-425</a>
should be available in HCatalog version 0.5.0.</p>
+
 </section>
 
 </section>



Mime
View raw message