hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koifman <ekoif...@hortonworks.com>
Subject Re: WebHCat and MapReduce JAR
Date Wed, 06 Aug 2014 20:03:33 GMT
can you try specify the path with "hdfs://...." otherwise it maybe looking
at local FS


On Wed, Aug 6, 2014 at 11:24 AM, Remy Dubois <rdubois@talend.com> wrote:

>  Hi again,
>
>
>
> And the issue I have is :
>
>
>
> 2014-08-06 22:06:37,083 FATAL [IPC Server handler 0 on 47378]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1407355085865_0004_m_000000_0 - exited :
> java.lang.ClassNotFoundException: routines.StringHandling
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>         at mapreduce.MyMapper.map(MyMapper.java:14)
>
>         at mapreduce.MyMapper.map(MyMapper.java:1)
>
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>
>         at
> org.apache.hadoop.mapreduce.lib.chain.Chain.runMapper(Chain.java:389)
>
>         at
> org.apache.hadoop.mapreduce.lib.chain.ChainMapper.run(ChainMapper.java:149)
>
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
> *De :* Remy Dubois [mailto:rdubois@talend.com]
> *Envoyé :* mercredi 6 août 2014 20:19
> *À :* user@hive.apache.org
> *Objet :* RE: WebHCat and MapReduce JAR
>
>
>
> Hi,
>
>
>
> Thanks a lot for your answer. Following some information about what I do.
>
>
>
> *WebHCat version*: Hive 0.13 embedded in HDP 2.1 GA.
>
>
>
> The webhcat logs don’t show anything interesting.
>
>
>
> *REST call:*
>
>
>
>
>
> *Jar location on HDFS:*
>
> *[hdfs@hdp21 root]$ hadoop fs -ls /tmp/systemRoutines.jar*
>
> -rw-r--r--   3 rdubois rdubois     264008 2014-08-06 19:16
> /tmp/systemRoutines.jar
>
> *[hdfs@hdp21 root]$ hadoop fs -get /tmp/systemRoutines.jar /home/hdfs*
>
> *[hdfs@hdp21 root]$ unzip -l /home/hdfs/systemRoutines.jar*
>
> Archive:  /home/hdfs/systemRoutines.jar
>
>   Length     Date   Time    Name
>
> --------    ----   ----    ----
>
>       102  08-06-14 14:39   META-INF/MANIFEST.MF
>
>      3609  08-06-14 14:39   routines/StringHandling.class
>
>
>
> Thanks,
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
> *De :* Eugene Koifman [mailto:ekoifman@hortonworks.com
> <ekoifman@hortonworks.com>]
> *Envoyé :* mercredi 6 août 2014 19:43
> *À :* user@hive.apache.org
> *Objet :* Re: WebHCat and MapReduce JAR
>
>
>
> libjars as form param should work.  Is there anything relevant in
> webhcat.log?  What version of webhcat are you using?  What is the exact
> REST call you are making?
>
>
>
> On Wed, Aug 6, 2014 at 10:17 AM, Remy Dubois <rdubois@talend.com> wrote:
>
> Hi,
>
>
>
> I’m trying to submit a MapReduce job through webhcat using the
> templeton/v1/mapreduce/jar path.
>
> My job depends on third party libraries. I have tried a lot of things such
> as using the libjars FormParam (the jar is located on HDFS), but it doesn’t
> work, I got ClassNotFoundException.
>
>
>
> I’ve workaround my issue merging the job jar and the dependencies jars,
> but doing that is not really possible with my use case.
>
>
>
> Can you please let me know if what I’m trying to do is possible ? I am
> able to make the same job work, using the ToolRunner API (with the –libjars
> parameter) without any problem.
>
>
>
> Thanks a lot in advance,
>
>
>
> *Remy Dubois*
> *Component team manager and Big Data principal *
> *Tel *+33 1 46 25 06 00 | *Mobile *+33 6 11 73 56 72 | *Email *
> rdubois@talend.com | *Skype *remy.dubois.talend
> *Talend* *SA *| 9, rue Pagès - 92150 Suresnes - France | +33 1 46 25 06 00
> | www.talend.com
> *Global Leader in Open Source Integration*
>
> [image: http://www.talend.com/sites/default/files/logo-talend.jpg]
> <http://www.talend.com/>
>
>
>
>
>
> --
>
>
>
> Thanks,
>
> Eugene
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 

Thanks,
Eugene

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message