incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Estes <james.es...@gmail.com>
Subject Re: Which jars to store to HCatalog in Pig?
Date Fri, 22 Jun 2012 16:38:11 GMT
I'm pretty sure 'register'ing isn't what you want.  I think you need to
specify the jars on the pig command line with:
-Dpig.additional.jars=...
see the "Running Pig with HCatalog" section here:
http://incubator.apache.org/hcatalog/docs/r0.4.0/loadstore.html

In short you need to set up a script that does the following:

export HADOOP_HOME=/usr/lib/hadoop
export HCAT_HOME=/usr/local/hcat
export HIVE_HOME=/usr/local/hive
export
PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/lib/hive-metastore-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar

export PIG_OPTS=-Dhive.metastore.uris=thrift://localhost:8787

pig -Dpig.additional.jars=$PIG_CLASSPATH

James

On Thu, Jun 21, 2012 at 10:11 PM, Russell Jurney
<russell.jurney@gmail.com>wrote:

> Now, I'm loading the required jars but not able to store to HCatalog.
>
> https://gist.github.com/2970165
>
> On Mon, Jun 11, 2012 at 6:36 PM, Russell Jurney <russell.jurney@gmail.com>wrote:
>
>> Much thanks!
>>
>>
>> On Mon, Jun 11, 2012 at 6:30 PM, Aniket Mokashi <aniket486@gmail.com>wrote:
>>
>>> Minimum set-
>>> /share/hcatalog/hcatalog*.jar:
>>> /share/hcatalog/lib/hive-metastore*.jar:
>>> /share/hcatalog/lib/thrift-fb303-*.jar:
>>> /share/hcatalog/lib/libthrift*.jar:
>>> /share/hcatalog/lib/hive-exec-*.jar
>>> +HCatalog configuration (hive-site et al)...
>>>
>>> These are all you need to make hive client to work.
>>>
>>> ~Aniket
>>>
>>> On Mon, Jun 11, 2012 at 4:55 PM, Vandana Ayyalasomayajula <
>>> avandana@yahoo-inc.com> wrote:
>>>
>>>> Hi Russel,
>>>>
>>>> I have the following jars and  hive-site.xml ( containing URI info to
>>>> connect to metastore) in the PIG_CLASSPATH to use Pig with HCatalog.
>>>>
>>>> guava-r09.jar
>>>> hbase-storage-handler-0.1.0.jar
>>>> hcatalog-0.5.0-dev.jar
>>>> hcatSupport-0.5.0-dev.jar
>>>> hive-builtins-<version>.jar
>>>> hive-cli-<version>.jar
>>>> hive-common-<version>.jar
>>>> hive-contrib-<version>.jar
>>>> hive-exec-<version>.jar
>>>> hive-hbase-handler-<version>.jar
>>>> hive-jdbc-<version>.jar
>>>> hive-metastore-<version>.jar
>>>> hive-pdk-<version>.jar
>>>> hive-serde-<version>.jar
>>>> hive-service-<version>.jar
>>>> hive-shims-<version>.jar
>>>> libfb303-<version>.jar
>>>>
>>>> Hope this helps.
>>>> Thanks
>>>> Vandana
>>>>
>>>>
>>>> On Jun 11, 2012, at 3:46 PM, Russell Jurney wrote:
>>>>
>>>> Which jars need to be loaded to store to HCatalog from Pig?  I'm
>>>> reading lots of docs:
>>>>
>>>> http://incubator.apache.org/hcatalog/docs/r0.2.0/loadstore.html
>>>>
>>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/api/org/apache/hcatalog/pig/package-summary.html
>>>> https://cwiki.apache.org/confluence/display/HCATALOG/How+To+Test
>>>>
>>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/04/hcatalog-tables-and-metadata-for-hadoop/
>>>> http://docs.hortonworks.com/HCatalog_Documentation/index.pdf
>>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/inputoutput.html
>>>>
>>>> But I can't find this, and there are a LOT of jars in HCatalog :)
>>>>
>>>> --
>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>>
>>>>
>>>> Vandana Ayyalasomayajula
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> "...:::Aniket:::... Quetzalco@tl"
>>>
>>
>>
>>
>> --
>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
>> com
>>
>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
> com
>

Mime
View raw message