From hcatalog-commits-return-498-apmail-incubator-hcatalog-commits-archive=incubator.apache.org@incubator.apache.org Sun Oct 2 21:07:00 2011 Return-Path: X-Original-To: apmail-incubator-hcatalog-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-hcatalog-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E09997AA2 for ; Sun, 2 Oct 2011 21:07:00 +0000 (UTC) Received: (qmail 82437 invoked by uid 500); 2 Oct 2011 21:07:00 -0000 Delivered-To: apmail-incubator-hcatalog-commits-archive@incubator.apache.org Received: (qmail 82412 invoked by uid 500); 2 Oct 2011 21:07:00 -0000 Mailing-List: contact hcatalog-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hcatalog-dev@incubator.apache.org Delivered-To: mailing list hcatalog-commits@incubator.apache.org Received: (qmail 82404 invoked by uid 99); 2 Oct 2011 21:07:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Oct 2011 21:07:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Oct 2011 21:06:56 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 6D2ED2388C73; Sun, 2 Oct 2011 21:05:41 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1178252 [27/34] - in /incubator/hcatalog/site: author/src/documentation/content/xdocs/ publish/docs/ publish/docs/r0.2.0/ publish/docs/r0.2.0/api/ publish/docs/r0.2.0/api/org/ publish/docs/r0.2.0/api/org/apache/ publish/docs/r0.2.0/api/org... Date: Sun, 02 Oct 2011 21:05:30 -0000 To: hcatalog-commits@incubator.apache.org From: hashutosh@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111002210541.6D2ED2388C73@eris.apache.org> Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximLoader.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximLoader.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximLoader.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximLoader.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.hcatalog.pig.HCatEximLoader (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.hcatalog.pig.HCatEximLoader

+
+No usage of org.apache.hcatalog.pig.HCatEximLoader +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+ + + Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximStorer.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximStorer.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximStorer.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatEximStorer.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.hcatalog.pig.HCatEximStorer (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.hcatalog.pig.HCatEximStorer

+
+No usage of org.apache.hcatalog.pig.HCatEximStorer +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+ + + Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatLoader.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatLoader.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatLoader.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatLoader.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.hcatalog.pig.HCatLoader (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.hcatalog.pig.HCatLoader

+
+No usage of org.apache.hcatalog.pig.HCatLoader +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+ + + Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatStorer.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatStorer.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatStorer.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/HCatStorer.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.hcatalog.pig.HCatStorer (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.hcatalog.pig.HCatStorer

+
+No usage of org.apache.hcatalog.pig.HCatStorer +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+ + + Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/PigHCatUtil.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/PigHCatUtil.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/PigHCatUtil.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/class-use/PigHCatUtil.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.hcatalog.pig.PigHCatUtil (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.hcatalog.pig.PigHCatUtil

+
+No usage of org.apache.hcatalog.pig.PigHCatUtil +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+ + + Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputDriver.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputDriver.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputDriver.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputDriver.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,493 @@ + + + + + + +LoadFuncBasedInputDriver (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.hcatalog.pig.drivers +
+Class LoadFuncBasedInputDriver

+
+java.lang.Object
+  extended by org.apache.hcatalog.mapreduce.HCatInputStorageDriver
+      extended by org.apache.hcatalog.pig.drivers.LoadFuncBasedInputDriver
+
+
+
Direct Known Subclasses:
PigStorageInputDriver
+
+
+
+
public abstract class LoadFuncBasedInputDriver
extends HCatInputStorageDriver
+ + +

+This is a base class which wraps a Load func in HCatInputStorageDriver. + If you already have a LoadFunc, then this class along with LoadFuncBasedInputFormat + is doing all the heavy lifting. For a new HCat Input Storage Driver just extend it + and override the initialize(). PigStorageInputDriver illustrates + that well. +

+ +

+

+ +

+ + + + + + + + + + + +
+Field Summary
+protected  org.apache.pig.LoadFunclf + +
+           
+  + + + + + + + + + + +
+Constructor Summary
LoadFuncBasedInputDriver() + +
+           
+  + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Method Summary
+ HCatRecordconvertToHCatRecord(org.apache.hadoop.io.WritableComparable baseKey, + org.apache.hadoop.io.Writable baseValue) + +
+          Converts to HCatRecord format usable by HCatInputFormat to convert to required valuetype.
+ org.apache.hadoop.mapreduce.InputFormat<? extends org.apache.hadoop.io.WritableComparable,? extends org.apache.hadoop.io.Writable>getInputFormat(java.util.Properties hcatProperties) + +
+          Returns the InputFormat to use with this Storage Driver.
+ voidinitialize(org.apache.hadoop.mapreduce.JobContext context, + java.util.Properties storageDriverArgs) + +
+           
+ voidsetInputPath(org.apache.hadoop.mapreduce.JobContext jobContext, + java.lang.String location) + +
+          Set the data location for the input.
+ voidsetOriginalSchema(org.apache.hadoop.mapreduce.JobContext jobContext, + HCatSchema hcatSchema) + +
+          Set the schema of the data as originally published in HCat.
+ voidsetOutputSchema(org.apache.hadoop.mapreduce.JobContext jobContext, + HCatSchema hcatSchema) + +
+          Set the consolidated schema for the HCatRecord data returned by the storage driver.
+ voidsetPartitionValues(org.apache.hadoop.mapreduce.JobContext jobContext, + java.util.Map<java.lang.String,java.lang.String> partitionValues) + +
+          Sets the partition key values for the current partition.
+  + + + + + + + +
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Field Detail
+ +

+lf

+
+protected org.apache.pig.LoadFunc lf
+
+
+
+
+ + + + + + + + +
+Constructor Detail
+ +

+LoadFuncBasedInputDriver

+
+public LoadFuncBasedInputDriver()
+
+
+ + + + + + + + +
+Method Detail
+ +

+convertToHCatRecord

+
+public HCatRecord convertToHCatRecord(org.apache.hadoop.io.WritableComparable baseKey,
+                                      org.apache.hadoop.io.Writable baseValue)
+                               throws java.io.IOException
+
+
Description copied from class: HCatInputStorageDriver
+
Converts to HCatRecord format usable by HCatInputFormat to convert to required valuetype. + Implementers of StorageDriver should look to overwriting this function so as to convert their + value type to HCatRecord. Default implementation is provided for StorageDriver implementations + on top of an underlying InputFormat that already uses HCatRecord as a tuple +

+

+
Specified by:
convertToHCatRecord in class HCatInputStorageDriver
+
+
+
baseValue - the underlying value to convert to HCatRecord +
Throws: +
java.io.IOException
+
+
+
+ +

+getInputFormat

+
+public org.apache.hadoop.mapreduce.InputFormat<? extends org.apache.hadoop.io.WritableComparable,? extends org.apache.hadoop.io.Writable> getInputFormat(java.util.Properties hcatProperties)
+
+
Description copied from class: HCatInputStorageDriver
+
Returns the InputFormat to use with this Storage Driver. +

+

+
Specified by:
getInputFormat in class HCatInputStorageDriver
+
+
+
Parameters:
hcatProperties - the properties containing parameters required for initialization of InputFormat +
Returns:
the InputFormat instance
+
+
+
+ +

+setOriginalSchema

+
+public void setOriginalSchema(org.apache.hadoop.mapreduce.JobContext jobContext,
+                              HCatSchema hcatSchema)
+                       throws java.io.IOException
+
+
Description copied from class: HCatInputStorageDriver
+
Set the schema of the data as originally published in HCat. The storage driver might validate that this matches with + the schema it has (like Zebra) or it will use this to create a HCatRecord matching the output schema. +

+

+
Specified by:
setOriginalSchema in class HCatInputStorageDriver
+
+
+
Parameters:
jobContext - the job context object
hcatSchema - the schema published in HCat for this data +
Throws: +
java.io.IOException - Signals that an I/O exception has occurred.
+
+
+
+ +

+setOutputSchema

+
+public void setOutputSchema(org.apache.hadoop.mapreduce.JobContext jobContext,
+                            HCatSchema hcatSchema)
+                     throws java.io.IOException
+
+
Description copied from class: HCatInputStorageDriver
+
Set the consolidated schema for the HCatRecord data returned by the storage driver. All tuples returned by the RecordReader should + have this schema. Nulls should be inserted for columns not present in the data. +

+

+
Specified by:
setOutputSchema in class HCatInputStorageDriver
+
+
+
Parameters:
jobContext - the job context object
hcatSchema - the schema to use as the consolidated schema +
Throws: +
java.io.IOException - Signals that an I/O exception has occurred.
+
+
+
+ +

+setPartitionValues

+
+public void setPartitionValues(org.apache.hadoop.mapreduce.JobContext jobContext,
+                               java.util.Map<java.lang.String,java.lang.String> partitionValues)
+                        throws java.io.IOException
+
+
Description copied from class: HCatInputStorageDriver
+
Sets the partition key values for the current partition. The storage driver is passed this so that the storage + driver can add the partition key values to the output HCatRecord if the partition key values are not present on disk. +

+

+
Specified by:
setPartitionValues in class HCatInputStorageDriver
+
+
+
Parameters:
jobContext - the job context object
partitionValues - the partition values having a map with partition key name as key and the HCatKeyValue as value +
Throws: +
java.io.IOException - Signals that an I/O exception has occurred.
+
+
+
+ +

+initialize

+
+public void initialize(org.apache.hadoop.mapreduce.JobContext context,
+                       java.util.Properties storageDriverArgs)
+                throws java.io.IOException
+
+
+
Overrides:
initialize in class HCatInputStorageDriver
+
+
+ +
Throws: +
java.io.IOException
+
+
+
+ +

+setInputPath

+
+public void setInputPath(org.apache.hadoop.mapreduce.JobContext jobContext,
+                         java.lang.String location)
+                  throws java.io.IOException
+
+
Description copied from class: HCatInputStorageDriver
+
Set the data location for the input. +

+

+
Overrides:
setInputPath in class HCatInputStorageDriver
+
+
+
Parameters:
jobContext - the job context object
location - the data location +
Throws: +
java.io.IOException - Signals that an I/O exception has occurred. + + Default implementation for FileInputFormat based Input Formats. Override + this for other input formats.
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ + + Added: incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.html URL: http://svn.apache.org/viewvc/incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.html?rev=1178252&view=auto ============================================================================== --- incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.html (added) +++ incubator/hcatalog/site/publish/docs/r0.2.0/api/org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.html Sun Oct 2 21:05:22 2011 @@ -0,0 +1,296 @@ + + + + + + +LoadFuncBasedInputFormat (HCatalog 0.2.0-incubating API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.hcatalog.pig.drivers +
+Class LoadFuncBasedInputFormat

+
+java.lang.Object
+  extended by org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.BytesWritable,org.apache.pig.data.Tuple>
+      extended by org.apache.hcatalog.pig.drivers.LoadFuncBasedInputFormat
+
+
+
+
public class LoadFuncBasedInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.BytesWritable,org.apache.pig.data.Tuple>
+ + +

+based on PigStorage +

+ +

+

+ +

+ + + + + + + + + + + +
+Constructor Summary
LoadFuncBasedInputFormat(org.apache.pig.LoadFunc loadFunc, + org.apache.pig.ResourceSchema dataSchema) + +
+           
+  + + + + + + + + + + + + + + + +
+Method Summary
+ org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.BytesWritable,org.apache.pig.data.Tuple>createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, + org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) + +
+           
+ java.util.List<org.apache.hadoop.mapreduce.InputSplit>getSplits(org.apache.hadoop.mapreduce.JobContext jobContext) + +
+           
+  + + + + + + + +
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Constructor Detail
+ +

+LoadFuncBasedInputFormat

+
+public LoadFuncBasedInputFormat(org.apache.pig.LoadFunc loadFunc,
+                                org.apache.pig.ResourceSchema dataSchema)
+
+
+ + + + + + + + +
+Method Detail
+ +

+createRecordReader

+
+public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.BytesWritable,org.apache.pig.data.Tuple> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
+                                                                                                                                 org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
+                                                                                                                          throws java.io.IOException,
+                                                                                                                                 java.lang.InterruptedException
+
+
+
Specified by:
createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.BytesWritable,org.apache.pig.data.Tuple>
+
+
+ +
Throws: +
java.io.IOException +
java.lang.InterruptedException
+
+
+
+ +

+getSplits

+
+public java.util.List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext jobContext)
+                                                                 throws java.io.IOException,
+                                                                        java.lang.InterruptedException
+
+
+
Specified by:
getSplits in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.BytesWritable,org.apache.pig.data.Tuple>
+
+
+ +
Throws: +
java.io.IOException +
java.lang.InterruptedException
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ + +