incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Which jars to store to HCatalog in Pig?
Date Fri, 22 Jun 2012 19:35:03 GMT
https://gist.github.com/2970165 has updated information after my
environment is setup.

On Fri, Jun 22, 2012 at 12:20 PM, Daniel Dai <daijy@hortonworks.com> wrote:

> If you can share your session, I'd happy to take a look.
>
> On Fri, Jun 22, 2012 at 12:14 PM, Russell Jurney <russell.jurney@gmail.com
> > wrote:
>
>> Thanks for your help, https://gist.github.com/2970165 now contains the
>> 'ls' result too.
>>
>> On Fri, Jun 22, 2012 at 10:14 AM, Travis Crawford <
>> traviscrawford@gmail.com> wrote:
>>
>>> I just hopped in #hcat on freenode too.
>>>
>>> --travis
>>>
>>>
>>> On Fri, Jun 22, 2012 at 10:12 AM, Travis Crawford
>>> <traviscrawford@gmail.com> wrote:
>>> > You shouldn't need to add jars on your classpath anymore; I allegedly
>>> > fixed this in PIG-2532.
>>> >
>>> > To double-check, what does "ls" of these globs show:
>>> >
>>> > register /usr/local/hcat/share/hcatalog/*.jar
>>> > register /home/hadoop/hive-0.9.0/lib/*.jar
>>> >
>>> > --travis
>>> >
>>> >
>>> > On Fri, Jun 22, 2012 at 9:38 AM, James Estes <james.estes@gmail.com>
>>> wrote:
>>> >> I'm pretty sure 'register'ing isn't what you want.  I think you need
>>> to
>>> >> specify the jars on the pig command line with:
>>> >> -Dpig.additional.jars=...
>>> >> see the "Running Pig with HCatalog" section here:
>>> >> http://incubator.apache.org/hcatalog/docs/r0.4.0/loadstore.html
>>> >>
>>> >> In short you need to set up a script that does the following:
>>> >>
>>> >> export HADOOP_HOME=/usr/lib/hadoop
>>> >> export HCAT_HOME=/usr/local/hcat
>>> >> export HIVE_HOME=/usr/local/hive
>>> >> export
>>> >>
>>> PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/lib/hive-metastore-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar
>>> >>
>>> >> export PIG_OPTS=-Dhive.metastore.uris=thrift://localhost:8787
>>> >>
>>> >> pig -Dpig.additional.jars=$PIG_CLASSPATH
>>> >>
>>> >> James
>>> >>
>>> >>
>>> >> On Thu, Jun 21, 2012 at 10:11 PM, Russell Jurney <
>>> russell.jurney@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Now, I'm loading the required jars but not able to store to HCatalog.
>>> >>>
>>> >>> https://gist.github.com/2970165
>>> >>>
>>> >>> On Mon, Jun 11, 2012 at 6:36 PM, Russell Jurney <
>>> russell.jurney@gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Much thanks!
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jun 11, 2012 at 6:30 PM, Aniket Mokashi <
>>> aniket486@gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Minimum set-
>>> >>>>> /share/hcatalog/hcatalog*.jar:
>>> >>>>> /share/hcatalog/lib/hive-metastore*.jar:
>>> >>>>> /share/hcatalog/lib/thrift-fb303-*.jar:
>>> >>>>> /share/hcatalog/lib/libthrift*.jar:
>>> >>>>> /share/hcatalog/lib/hive-exec-*.jar
>>> >>>>> +HCatalog configuration (hive-site et al)...
>>> >>>>>
>>> >>>>> These are all you need to make hive client to work.
>>> >>>>>
>>> >>>>> ~Aniket
>>> >>>>>
>>> >>>>> On Mon, Jun 11, 2012 at 4:55 PM, Vandana Ayyalasomayajula
>>> >>>>> <avandana@yahoo-inc.com> wrote:
>>> >>>>>>
>>> >>>>>> Hi Russel,
>>> >>>>>>
>>> >>>>>> I have the following jars and  hive-site.xml ( containing
URI
>>> info to
>>> >>>>>> connect to metastore) in the PIG_CLASSPATH to use Pig
with
>>> HCatalog.
>>> >>>>>>
>>> >>>>>> guava-r09.jar
>>> >>>>>> hbase-storage-handler-0.1.0.jar
>>> >>>>>> hcatalog-0.5.0-dev.jar
>>> >>>>>> hcatSupport-0.5.0-dev.jar
>>> >>>>>> hive-builtins-<version>.jar
>>> >>>>>> hive-cli-<version>.jar
>>> >>>>>> hive-common-<version>.jar
>>> >>>>>> hive-contrib-<version>.jar
>>> >>>>>> hive-exec-<version>.jar
>>> >>>>>> hive-hbase-handler-<version>.jar
>>> >>>>>> hive-jdbc-<version>.jar
>>> >>>>>> hive-metastore-<version>.jar
>>> >>>>>> hive-pdk-<version>.jar
>>> >>>>>> hive-serde-<version>.jar
>>> >>>>>> hive-service-<version>.jar
>>> >>>>>> hive-shims-<version>.jar
>>> >>>>>> libfb303-<version>.jar
>>> >>>>>>
>>> >>>>>> Hope this helps.
>>> >>>>>> Thanks
>>> >>>>>> Vandana
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Jun 11, 2012, at 3:46 PM, Russell Jurney wrote:
>>> >>>>>>
>>> >>>>>> Which jars need to be loaded to store to HCatalog from
Pig?  I'm
>>> >>>>>> reading lots of docs:
>>> >>>>>>
>>> >>>>>> http://incubator.apache.org/hcatalog/docs/r0.2.0/loadstore.html
>>> >>>>>>
>>> >>>>>>
>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/api/org/apache/hcatalog/pig/package-summary.html
>>> >>>>>> https://cwiki.apache.org/confluence/display/HCATALOG/How+To+Test
>>> >>>>>>
>>> >>>>>>
>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/04/hcatalog-tables-and-metadata-for-hadoop/
>>> >>>>>> http://docs.hortonworks.com/HCatalog_Documentation/index.pdf
>>> >>>>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/inputoutput.html
>>> >>>>>>
>>> >>>>>> But I can't find this, and there are a LOT of jars in
HCatalog :)
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Russell
>>> >>>>>> Jurney twitter.com/rjurney russell.jurney@gmail.com
>>> datasyndrome.com
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> Vandana Ayyalasomayajula
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> "...:::Aniket:::... Quetzalco@tl"
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Russell
>>> >>>> Jurney twitter.com/rjurney russell.jurney@gmail.com
>>> datasyndrome.com
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Russell
>>> >>> Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com
>>> >>
>>> >>
>>>
>>
>>
>>
>> --
>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
>> com
>>
>
>


-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

Mime
View raw message