pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2786) enhance Pig launcher script wrt. HBase/HCat integration
Date Sat, 20 Apr 2013 17:49:15 GMT

    [ https://issues.apache.org/jira/browse/PIG-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637311#comment-13637311
] 

Daniel Dai commented on PIG-2786:
---------------------------------

Took a deeper look. We do ship jars to backend. The magic is in TableMapReduceUtil.addDependencyJars:
{code}
TableMapReduceUtil.addDependencyJars(job.getConfiguration(),
            org.apache.hadoop.hbase.client.HTable.class,
            com.google.common.collect.Lists.class,
            org.apache.zookeeper.ZooKeeper.class)
{code}

However, we didn't ship protobuf.jar and it is used in latest hbase. HBaseHCatStorageHandler
works because it ship protobuf.jar.

So with this patch, HBaseHCatStorageHandler works, plus we add right hbase jars in the CLASSPATH
(if set HBASE_HOME correctly). For shipping protobuf.jar in HBaseStorage, we will track in
PIG-3285.
                
> enhance Pig launcher script wrt. HBase/HCat integration
> -------------------------------------------------------
>
>                 Key: PIG-2786
>                 URL: https://issues.apache.org/jira/browse/PIG-2786
>             Project: Pig
>          Issue Type: Improvement
>          Components: grunt
>    Affects Versions: 0.10.0
>            Reporter: Roman Shaposhnik
>            Assignee: Roman Shaposhnik
>            Priority: Minor
>              Labels: hbase
>         Attachments: 0001-PIG-2786-launch-script-should-locate-HBase.patch
>
>
> The current bin/pig script suffers from a couple of issues as far as integration with
HBase is concerned:
>   # it only detects ZK/HBase jars under a PIG_HOME/share/.. layout
>   # it doesn't detect HBase dependencies
> The proposal here would be to ask HBase itself for its classpath

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message