Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 66059104AE for ; Sun, 4 Jan 2015 16:36:11 +0000 (UTC) Received: (qmail 73415 invoked by uid 500); 4 Jan 2015 16:35:59 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 73251 invoked by uid 500); 4 Jan 2015 16:35:59 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 73240 invoked by uid 99); 4 Jan 2015 16:35:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2015 16:35:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [209.85.218.51] (HELO mail-oi0-f51.google.com) (209.85.218.51) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2015 16:35:33 +0000 Received: by mail-oi0-f51.google.com with SMTP id h136so11583771oig.10 for ; Sun, 04 Jan 2015 08:32:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=XnL0lQtYTEMo9RvmU4VGFXxgCIfQYaI3VdR2SHT3XmE=; b=jFIuOPt9iB2XcHZH6UboKUQnLUwPf/iVPi6XRixDkcWQbwOZd8pDLDSg7mp6ZMEhxN HLxMh9NwPPVSlomx3C5ZBIXYFWo6JDrsp8bryzemraMKgd92cBk2M4YydMYBIJBTrihI Io1z5nTLNBe/Ihx9TEtrMCb6TEs8gclO0Yxr712gjtlXaZgu15CI5LRGC0RKnDoNXBfq 3tPEvW8moORm0w69RlqoGJlsEt9ph5x64w76woi69VBrCtbbHNgSnVKHz+FCUHFCJ/Jz qPHFkCE4y1rsWPI4r1CQQqALr7YknSsnhlGzzBzz28v+a9eLWro5lTQBYfThu+yDPglt O2JQ== X-Gm-Message-State: ALoCoQlDPLwaQsxFyQPJQ6grBbPm4e/tQa0o1x4Pwrvzd/KS1mkyynxsMXpgRZYyY3FoGCzJXNqv MIME-Version: 1.0 X-Received: by 10.202.189.2 with SMTP id n2mr38236162oif.4.1420389176472; Sun, 04 Jan 2015 08:32:56 -0800 (PST) Sender: niels@basj.es Received: by 10.76.150.41 with HTTP; Sun, 4 Jan 2015 08:32:56 -0800 (PST) X-Originating-IP: [2001:980:91c0:1:61ce:595d:d6a8:5f66] In-Reply-To: References: Date: Sun, 4 Jan 2015 17:32:56 +0100 X-Google-Sender-Auth: 5kB5olscplfzuFUwL0ah6s9JkZw Message-ID: Subject: Re: way to add custom udf jar in hadoop 2.x version From: Niels Basjes To: user@hadoop.apache.org Cc: "dev@hive.apache.org" Content-Type: multipart/alternative; boundary=001a113ddd9c4e8b91050bd62075 X-Virus-Checked: Checked by ClamAV on apache.org --001a113ddd9c4e8b91050bd62075 Content-Type: text/plain; charset=UTF-8 I created https://issues.apache.org/jira/browse/HIVE-9252 for this improvement. On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes wrote: > Hi, > > These options: > - HIVE_HOME/auxlib > - http://stackoverflow.com/questions/14032924/how-to-add-serde-jar > - ADD JAR commands in your $HOME/.hiverc file > > either require IT operations to put my JAR on all nodes OR I cannot share > it, Only works on the commandline and it won't work in HUE/Beeswax. > > Now "Permanent Functions": > - https://issues.apache.org/jira/browse/HIVE-6047 > - > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions > > What these "Permanent Functions" do is: > 1) put the jar on the cluster without IT operations putting the jar on all > nodes > 2) the jar is used transparently for everyone who want to use the function. > > I am writing a deserializer [1] (Not finished yet: > https://github.com/nielsbasjes/logparser/blob/master/README-Hive.md) that > should make existing files query-able as an external table in Hive. > > Question is: Is there something similar for CREATE EXTERNAL TABLE ?? > > Something like > > CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name > ... > STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)] > [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ]; > > > Is this something for which there is already a JIRA (couldn't find it)? > If not; Should I create one? (I.e. do you think this would make sense for > others?) > > Niels Basjes > > > On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey < > Alexey.Yakubovich@searshc.com> wrote: > >> Try to look hr: >> http://stackoverflow.com/questions/14032924/how-to-add-serde-jar >> >> Another advice: insert your ADD JAR commands in your $HOME/.hiverc file >> and start hive. ( >> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH595qZidjEQ@mail.gmail.com%3E >> ) >> >> >> >> From: Ted Yu >> Reply-To: "user@hadoop.apache.org" >> Date: Wednesday, December 31, 2014 at 8:25 AM >> To: "dev@hive.apache.org" >> Subject: Fwd: way to add custom udf jar in hadoop 2.x version >> >> Forwarding Niels' question to hive mailing list. >> >> On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes wrote: >> >>> Thanks for the pointer. >>> This seems to work for functions. Is there something similar for CREATE >>> EXTERNAL TABLE ?? >>> >>> Niels >>> On Dec 31, 2014 8:13 AM, "Ted Yu" wrote: >>> >>>> Have you seen this thread ? >>>> >>>> http://search-hadoop.com/m/8er9TcALc/Hive+udf+custom+jar&subj=Best+way+to+add+custom+UDF+jar+in+HiveServer2 >>>> >>>> On Dec 30, 2014, at 10:56 PM, reena upadhyay >>>> wrote: >>>> >>>> Hi, >>>> >>>> I am using hadoop 2.4.0 version. I have created custom udf jar. I am >>>> trying to execute a simple select udf query using java hive jdbc client >>>> program. When hive execute the query using map reduce job, then the query >>>> execution get fails because the mapper is not able to locate the udf class. >>>> So I wanted to add the udf jar in hadoop environment permanently. >>>> Please suggest me a way to add this external jar for single node and multi >>>> node hadoop cluster. >>>> >>>> PS: I am using hive 0.13.1 version and I already have this custom udf >>>> jar added in HIVE_HOME/lib directory. >>>> >>>> >>>> Thanks >>>> >>>> >> This message, including any attachments, is the property of Sears >> Holdings Corporation and/or one of its subsidiaries. It is confidential and >> may contain proprietary or legally privileged information. If you are not >> the intended recipient, please delete it without reading the contents. >> Thank you. >> > > > > -- > Best regards / Met vriendelijke groeten, > > Niels Basjes > -- Best regards / Met vriendelijke groeten, Niels Basjes --001a113ddd9c4e8b91050bd62075 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I created=C2=A0https://issues.apache.org/jira/browse/HIVE-9252 for th= is improvement.

On Sun, Jan 4, 2015 at 5:16 PM, Niels Basjes <Niels@basjes.nl> wrote:
Hi,

These options:
-=C2=A0HIVE_HOME/auxlib
- ADD JAR commands in your $HO= ME/.hiverc file

either require I= T operations to put my JAR on all nodes OR I cannot share it, Only works on= the commandline and it won't work in HUE/Beeswax.

=
Now "Permanent Functions":
-=C2=A0https://cwiki.apache.org/confluence= /display/Hive/LanguageManual+DDL#LanguageManualDDL-PermanentFunctions

Wh= at these "Permanent Functions" do is:
1) put the jar on= the cluster without IT operations putting the jar on all nodes
2) the jar is used transparently for everyon= e who want to use the function.

<= div class=3D"gmail_extra">I am writing a deserializer [1] (Not=C2=A0finishe= d yet: https://github.com/nielsbasjes/logparser/blob= /master/README-Hive.md) that should make existing files query-able as a= n external table in Hive.

Question is:=C2=A0Is there something similar for CREATE EXTER= NAL TABLE ??

Something like=C2=A0

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EX= ISTS] [db_name.]table_name
=C2=A0 =C2= =A0 ...
=C2=A0 =C2=A0 STORED BY 'storage.handler.c= lass.name' [WITH SERDEPROPERTIES (...)]=C2=A0
=C2=A0 =C2=A0 [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|= FILE|ARCHIVE 'file_uri'] ];

Is t= his something for which there is already a JIRA (couldn't find it)?
If not; Should I create one? (I.e. do you thin= k this would make sense for others?)

Niels Basjes


On Fri, Jan 2, 2015 at 9:00 PM, Yakubovich, Alexey <Alexey.Yakubovich@searshc.com> wrote:

Another advice:=C2=A0insert your ADD JAR commands in your $HOME/.hiverc fi= le and start hive. (http://mail-archives.apache.org/= mod_mbox/hive-user/201303.mbox/%3CCAMGr+0h3SMDw4zHTpYo5B1B4iob05BPW8LS+dAEH= 595qZidjEQ@mail.gmail.com%3E)



From: Ted Yu <yuzhihong@gmail.com>
Reply-To: "user@hadoop.apache.org" &= lt;user@hadoop.= apache.org>
Date: Wednesday, December 31, 2014 = at 8:25 AM
To: "dev@hive.apache.org" <dev@hive.apache.org&g= t;
Subject: Fwd: way to add custom udf= jar in hadoop 2.x version

Forwarding Niels' question to hive mailing l= ist.

On Wed, Dec 31, 2014 at 1:24 AM, Niels Basjes <Niels@basjes.nl> wrote:

Thanks for the pointer.
This seems to work for functions. Is there something similar for CREATE EXT= ERNAL TABLE ??

Niels
On Dec 31, 2014 8:13 AM, "Ted Yu" <= yuzhihong@gmail.co= m> wrote:
Have you seen this thread ?

On Dec 30, 2014, at 10:56 PM, reena upadhyay <reena2485@gmail.com> wrote:

Hi,

I am using hadoop 2.4.0 version. I have created custom udf jar. I am trying= to execute a simple select udf query using java hive jdbc client program. = When hive execute the query using map reduce job, then the query execution = get fails because the mapper is not able to locate the udf class.
So I wanted to add the udf jar in hadoop environment permanently. Plea= se suggest me a way to add this external jar for single node and multi node= hadoop cluster.

PS: I am using hive 0.13.1 version and I already have this custom udf = jar added in HIVE_HOME/lib directory.


Thanks
This message, including any attachments, is the property of Sears Holdings = Corporation and/or one of its subsidiaries. It is confidential and may cont= ain proprietary or legally privileged information. If you are not the inten= ded recipient, please delete it without reading the contents. Thank you.



--
Best regards / Met vrie= ndelijke groeten,

Niels Basjes



--
Best regards / Met vriendelijke groeten,

Niels = Basjes
--001a113ddd9c4e8b91050bd62075--