hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu A <hadoophi...@gmail.com>
Subject Re: Custom MR scripts using java in Hive
Date Wed, 26 Sep 2012 09:31:14 GMT
Hi Manish,
Thanks,I did like the same.but how to invoke the custom java map/reduce
functions ( com.hive.test.TestMapper ) since there is no script as it is a
jar file.The process looks bit different from UDF( used create temporary
function).


On Wed, Sep 26, 2012 at 12:25 PM, Manish.Bhoge <Manish.Bhoge@target.com>wrote:

>  Sorry for late reply. ****
>
> ** **
>
> For anything which you want to run as MAP and REDUCE you have to extend
> map reduce classes for your functionality irrespective of language (Java,
> python or any other).  Once you have extended class move the jar to the
> Hadoop cluster. ****
>
> Bertrand has also mention about reflection. That is something new for me.
> You can give a try to reflection.****
>
> ** **
>
> Thank You,****
>
> Manish****
>
> ** **
>
> *From:* Tamil A [mailto:4tamils04@gmail.com]
> *Sent:* Tuesday, September 25, 2012 6:48 PM
> *To:* user@hive.apache.org
> *Subject:* Re: Custom MR scripts using java in Hive****
>
> ** **
>
> Hi Manish,****
>
>  ****
>
> Thanks for your help.I did the same using UDF.Now trying with
> Transform,Map and Reduce clauses.so is it mean by using java we have to
> goahead through UDF and for other languages using MapReduce Scripts i.e.,
> the Transform,Map and Reduce clauses.****
>
> Please correct me if am wrong.****
>
>  ****
>
>  ****
>
>  ****
>
> Thanks & Regards,****
>
> Manu ****
>
> ** **
>
> On Tue, Sep 25, 2012 at 5:19 PM, Manish.Bhoge <Manish.Bhoge@target.com>
> wrote:****
>
> Manu,****
>
>  ****
>
> If you have written UDF in Java for Hive then you need to copy your JAR on
> your Hadoop cluster in /usr/lib/hive/lib/ folder to hive to use this JAR.
> ****
>
>  ****
>
> Thank You,****
>
> Manish****
>
>  ****
>
> *From:* Manu A [mailto:hadoophive4@gmail.com]
> *Sent:* Tuesday, September 25, 2012 3:44 PM
> *To:* user@hive.apache.org
> *Subject:* Custom MR scripts using java in Hive****
>
>  ****
>
> Hi All,****
>
> I am learning hive. Please let me know if any one tried with custom Map
> Reduce scripts using java in hive or refer me some links and blogs with an
> example.****
>
>  ****
>
> when i tried i got the below error :****
>
>  ****
>
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2012-09-25 02:47:23,720 Stage-1 map = 0%,  reduce = 0%
> 2012-09-25 02:47:56,943 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201209222231_0001 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_201209222231_0001_m_000002 (and more) from job
> job_201209222231_0001
> Exception in thread "Thread-51" java.lang.RuntimeException: Error while
> reading from task log url
>         at
> org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
>         at
> org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211)
>         at
> org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Server returned HTTP response code: 400
> for URL: // removed as confidential****
>
>         at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1313)
>         at java.net.URL.openStream(URL.java:1010)
>         at
> org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
>         ... 3 more
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> Thanks for ur help in advance :)****
>
>  ****
>
>  ****
>
>  ****
>
> Thanks & Regards,****
>
> Manu ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>
>
>
> --
> *Thanks & Regards,* ****
>
> *Tamil*****
>
> ** **
>

Mime
View raw message