incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <da...@hortonworks.com>
Subject Re: Which jars to store to HCatalog in Pig?
Date Fri, 22 Jun 2012 21:21:55 GMT
It seems we still need PIG_CLASSPATH if we use HCat on store side.

I have a one line change to make it work:

Index: src/org/apache/pig/impl/PigContext.java
===================================================================
--- src/org/apache/pig/impl/PigContext.java    (revision 1351194)
+++ src/org/apache/pig/impl/PigContext.java    (working copy)
@@ -273,6 +273,7 @@
         if (resource != null) {
             extraJars.add(resource);
             PigContext.classloader = createCl(null);
+
Thread.currentThread().setContextClassLoader(PigContext.classloader);
         }
     }

Daniel

On Fri, Jun 22, 2012 at 12:35 PM, Russell Jurney
<russell.jurney@gmail.com>wrote:

> https://gist.github.com/2970165 has updated information after my
> environment is setup.
>
>
> On Fri, Jun 22, 2012 at 12:20 PM, Daniel Dai <daijy@hortonworks.com>wrote:
>
>> If you can share your session, I'd happy to take a look.
>>
>> On Fri, Jun 22, 2012 at 12:14 PM, Russell Jurney <
>> russell.jurney@gmail.com> wrote:
>>
>>> Thanks for your help, https://gist.github.com/2970165 now contains the
>>> 'ls' result too.
>>>
>>> On Fri, Jun 22, 2012 at 10:14 AM, Travis Crawford <
>>> traviscrawford@gmail.com> wrote:
>>>
>>>> I just hopped in #hcat on freenode too.
>>>>
>>>> --travis
>>>>
>>>>
>>>> On Fri, Jun 22, 2012 at 10:12 AM, Travis Crawford
>>>> <traviscrawford@gmail.com> wrote:
>>>> > You shouldn't need to add jars on your classpath anymore; I allegedly
>>>> > fixed this in PIG-2532.
>>>> >
>>>> > To double-check, what does "ls" of these globs show:
>>>> >
>>>> > register /usr/local/hcat/share/hcatalog/*.jar
>>>> > register /home/hadoop/hive-0.9.0/lib/*.jar
>>>> >
>>>> > --travis
>>>> >
>>>> >
>>>> > On Fri, Jun 22, 2012 at 9:38 AM, James Estes <james.estes@gmail.com>
>>>> wrote:
>>>> >> I'm pretty sure 'register'ing isn't what you want.  I think you
need
>>>> to
>>>> >> specify the jars on the pig command line with:
>>>> >> -Dpig.additional.jars=...
>>>> >> see the "Running Pig with HCatalog" section here:
>>>> >> http://incubator.apache.org/hcatalog/docs/r0.4.0/loadstore.html
>>>> >>
>>>> >> In short you need to set up a script that does the following:
>>>> >>
>>>> >> export HADOOP_HOME=/usr/lib/hadoop
>>>> >> export HCAT_HOME=/usr/local/hcat
>>>> >> export HIVE_HOME=/usr/local/hive
>>>> >> export
>>>> >>
>>>> PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/lib/hive-metastore-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar
>>>> >>
>>>> >> export PIG_OPTS=-Dhive.metastore.uris=thrift://localhost:8787
>>>> >>
>>>> >> pig -Dpig.additional.jars=$PIG_CLASSPATH
>>>> >>
>>>> >> James
>>>> >>
>>>> >>
>>>> >> On Thu, Jun 21, 2012 at 10:11 PM, Russell Jurney <
>>>> russell.jurney@gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Now, I'm loading the required jars but not able to store to
>>>> HCatalog.
>>>> >>>
>>>> >>> https://gist.github.com/2970165
>>>> >>>
>>>> >>> On Mon, Jun 11, 2012 at 6:36 PM, Russell Jurney <
>>>> russell.jurney@gmail.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Much thanks!
>>>> >>>>
>>>> >>>>
>>>> >>>> On Mon, Jun 11, 2012 at 6:30 PM, Aniket Mokashi <
>>>> aniket486@gmail.com>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> Minimum set-
>>>> >>>>> /share/hcatalog/hcatalog*.jar:
>>>> >>>>> /share/hcatalog/lib/hive-metastore*.jar:
>>>> >>>>> /share/hcatalog/lib/thrift-fb303-*.jar:
>>>> >>>>> /share/hcatalog/lib/libthrift*.jar:
>>>> >>>>> /share/hcatalog/lib/hive-exec-*.jar
>>>> >>>>> +HCatalog configuration (hive-site et al)...
>>>> >>>>>
>>>> >>>>> These are all you need to make hive client to work.
>>>> >>>>>
>>>> >>>>> ~Aniket
>>>> >>>>>
>>>> >>>>> On Mon, Jun 11, 2012 at 4:55 PM, Vandana Ayyalasomayajula
>>>> >>>>> <avandana@yahoo-inc.com> wrote:
>>>> >>>>>>
>>>> >>>>>> Hi Russel,
>>>> >>>>>>
>>>> >>>>>> I have the following jars and  hive-site.xml ( containing
URI
>>>> info to
>>>> >>>>>> connect to metastore) in the PIG_CLASSPATH to use
Pig with
>>>> HCatalog.
>>>> >>>>>>
>>>> >>>>>> guava-r09.jar
>>>> >>>>>> hbase-storage-handler-0.1.0.jar
>>>> >>>>>> hcatalog-0.5.0-dev.jar
>>>> >>>>>> hcatSupport-0.5.0-dev.jar
>>>> >>>>>> hive-builtins-<version>.jar
>>>> >>>>>> hive-cli-<version>.jar
>>>> >>>>>> hive-common-<version>.jar
>>>> >>>>>> hive-contrib-<version>.jar
>>>> >>>>>> hive-exec-<version>.jar
>>>> >>>>>> hive-hbase-handler-<version>.jar
>>>> >>>>>> hive-jdbc-<version>.jar
>>>> >>>>>> hive-metastore-<version>.jar
>>>> >>>>>> hive-pdk-<version>.jar
>>>> >>>>>> hive-serde-<version>.jar
>>>> >>>>>> hive-service-<version>.jar
>>>> >>>>>> hive-shims-<version>.jar
>>>> >>>>>> libfb303-<version>.jar
>>>> >>>>>>
>>>> >>>>>> Hope this helps.
>>>> >>>>>> Thanks
>>>> >>>>>> Vandana
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Jun 11, 2012, at 3:46 PM, Russell Jurney wrote:
>>>> >>>>>>
>>>> >>>>>> Which jars need to be loaded to store to HCatalog
from Pig?  I'm
>>>> >>>>>> reading lots of docs:
>>>> >>>>>>
>>>> >>>>>> http://incubator.apache.org/hcatalog/docs/r0.2.0/loadstore.html
>>>> >>>>>>
>>>> >>>>>>
>>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/api/org/apache/hcatalog/pig/package-summary.html
>>>> >>>>>> https://cwiki.apache.org/confluence/display/HCATALOG/How+To+Test
>>>> >>>>>>
>>>> >>>>>>
>>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/04/hcatalog-tables-and-metadata-for-hadoop/
>>>> >>>>>> http://docs.hortonworks.com/HCatalog_Documentation/index.pdf
>>>> >>>>>>
>>>> http://incubator.apache.org/hcatalog/docs/r0.4.0/inputoutput.html
>>>> >>>>>>
>>>> >>>>>> But I can't find this, and there are a LOT of jars
in HCatalog :)
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> Russell
>>>> >>>>>> Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> Vandana Ayyalasomayajula
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> "...:::Aniket:::... Quetzalco@tl"
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Russell
>>>> >>>> Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Russell
>>>> >>> Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>> >>
>>>> >>
>>>>
>>>
>>>
>>>
>>> --
>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome
>>> .com
>>>
>>
>>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
> com
>

Mime
View raw message