hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Re: way to add custom udf jar in hadoop 2.x version
Date Sun, 04 Jan 2015 16:16:04 GMT
Hi,

These options:
- HIVE_HOME/auxlib
- http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
- ADD JAR commands in your $HOME/.hiverc file

either require IT operations to put my JAR on all nodes OR I cannot share
it, Only works on the commandline and it won't work in HUE/Beeswax.

Now "Permanent Functions":
- https://issues.apache.org/jira/browse/HIVE-6047
-
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions

What these "Permanent Functions" do is:
1) put the jar on the cluster without IT operations putting the jar on all
nodes
2) the jar is used transparently for everyone who want to use the function.

I am writing a deserializer [1] (Not finished yet:
https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
should make existing files query-able as an external table in Hive.

Question is: Is there something similar for CREATE EXTERNAL TABLE ??

Something like

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
    ...
    STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
    [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];


Is this something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you think this would make sense for
others?)

Niels Basjes


On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
Alexey.Yakubovich@searshc.com> wrote:

>  Try to  look hr:
> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>
>  Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
> and start hive. (
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
> )
>
>
>
>   From: Ted Yu <yuzhihong@gmail.com>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Date: Wednesday, December 31, 2014 at 8:25 AM
> To: "dev@hive.apache.org" <dev@hive.apache.org>
> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>
>   Forwarding Niels' question to hive mailing list.
>
> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Niels@basjes.nl> wrote:
>
>> Thanks for the pointer.
>> This seems to work for functions. Is there something similar for CREATE
>> EXTERNAL TABLE ??
>>
>> Niels
>>   On Dec 31, 2014 8:13 AM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>>
>>>  Have you seen this thread ?
>>>
>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>
>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <reena2485@gmail.com>
>>> wrote:
>>>
>>>   Hi,
>>>
>>>  I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>> trying to execute a simple select udf query using java hive jdbc client
>>> program. When hive execute the query using map reduce job, then the query
>>> execution get fails because the mapper is not able to locate the udf class.
>>> So I wanted to add the udf jar in hadoop environment permanently. Please
>>> suggest me a way to add this external jar for single node and multi node
>>> hadoop cluster.
>>>
>>>  PS: I am using hive 0.13.1 version and I already have this custom udf
>>> jar added in HIVE_HOME/lib directory.
>>>
>>>
>>>  Thanks
>>>
>>>
>    This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential and
> may contain proprietary or legally privileged information. If you are not
> the intended recipient, please delete it without reading the contents.
> Thank you.
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Mime
View raw message