hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Re: way to add custom udf jar in hadoop 2.x version
Date Sun, 04 Jan 2015 16:32:56 GMT
I created https://issues.apache.org/jira/browse/HIVE-9252 for this
improvement.

On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <Niels@basjes.nl> wrote:

> Hi,
>
> These options:
> - HIVE_HOME/auxlib
> - http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
> - ADD JAR commands in your $HOME/.hiverc file
>
> either require IT operations to put my JAR on all nodes OR I cannot share
> it, Only works on the commandline and it won't work in HUE/Beeswax.
>
> Now "Permanent Functions":
> - https://issues.apache.org/jira/browse/HIVE-6047
> -
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions
>
> What these "Permanent Functions" do is:
> 1) put the jar on the cluster without IT operations putting the jar on all
> nodes
> 2) the jar is used transparently for everyone who want to use the function.
>
> I am writing a deserializer [1] (Not finished yet:
> https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that
> should make existing files query-able as an external table in Hive.
>
> Question is: Is there something similar for CREATE EXTERNAL TABLE ??
>
> Something like
>
> CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
>     ...
>     STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
>     [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
>
>
> Is this something for which there is already a JIRA (couldn't find it)?
> If not; Should I create one? (I.e. do you think this would make sense for
> others?)
>
> Niels Basjes
>
>
> On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <
> Alexey.Yakubovich@searshc.com> wrote:
>
>>  Try to  look hr:
>> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar
>>
>>  Another advice: insert your ADD JAR commands in your $HOME/.hiverc file
>> and start hive. (
>> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E
>> )
>>
>>
>>
>>   From: Ted Yu <yuzhihong@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Date: Wednesday, December 31, 2014 at 8:25 AM
>> To: "dev@hive.apache.org" <dev@hive.apache.org>
>> Subject: Fwd: way to add custom udf jar in hadoop 2.x version
>>
>>   Forwarding Niels' question to hive mailing list.
>>
>> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Niels@basjes.nl> wrote:
>>
>>> Thanks for the pointer.
>>> This seems to work for functions. Is there something similar for CREATE
>>> EXTERNAL TABLE ??
>>>
>>> Niels
>>>   On Dec 31, 2014 8:13 AM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>>>
>>>>  Have you seen this thread ?
>>>>
>>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2
>>>>
>>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay <reena2485@gmail.com>
>>>> wrote:
>>>>
>>>>   Hi,
>>>>
>>>>  I am using hadoop 2.4.0 version. I have created custom udf jar. I am
>>>> trying to execute a simple select udf query using java hive jdbc client
>>>> program. When hive execute the query using map reduce job, then the query
>>>> execution get fails because the mapper is not able to locate the udf class.
>>>> So I wanted to add the udf jar in hadoop environment permanently.
>>>> Please suggest me a way to add this external jar for single node and multi
>>>> node hadoop cluster.
>>>>
>>>>  PS: I am using hive 0.13.1 version and I already have this custom udf
>>>> jar added in HIVE_HOME/lib directory.
>>>>
>>>>
>>>>  Thanks
>>>>
>>>>
>>    This message, including any attachments, is the property of Sears
>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>> may contain proprietary or legally privileged information. If you are not
>> the intended recipient, please delete it without reading the contents.
>> Thank you.
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Mime
View raw message