Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 88D6918DCA for ; Fri, 18 Dec 2015 03:07:32 +0000 (UTC) Received: (qmail 2079 invoked by uid 500); 18 Dec 2015 03:07:28 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 1990 invoked by uid 500); 18 Dec 2015 03:07:28 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 1978 invoked by uid 99); 18 Dec 2015 03:07:28 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Dec 2015 03:07:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E06A1C8B52 for ; Fri, 18 Dec 2015 03:07:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.99 X-Spam-Level: ** X-Spam-Status: No, score=2.99 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id GCEWkqH-6FfS for ; Fri, 18 Dec 2015 03:07:19 +0000 (UTC) Received: from mail.meilishuo.com (mail.meilishuo.com [124.202.144.6]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTP id 7AF852051D for ; Fri, 18 Dec 2015 03:07:18 +0000 (UTC) Received: from BMRZMYNHH7SGJV9 (unknown [115.182.230.163]) (Authenticated sender: jipengzeng@meilishuo.com) by mail.meilishuo.com (Postfix) with ESMTPA id 78DE71AAA4F62; Fri, 18 Dec 2015 11:07:09 +0800 (CST) Date: Fri, 18 Dec 2015 11:07:10 +0800 From: "jipengzeng@meilishuo.com" To: user Cc: furcy.pin Subject: Re: Discussion: permanent UDF with database name References: X-Priority: 3 X-Has-Attach: no X-Mailer: Foxmail 7, 2, 7, 21[cn] Mime-Version: 1.0 Message-ID: <2015121811071013346335@meilishuo.com> Content-Type: multipart/alternative; boundary="----=_001_NextPart136843422704_=----" This is a multi-part message in MIME format. ------=_001_NextPart136843422704_=---- Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 QCBGdXJjeSBQaW4NCkkgYWdyZWUgeW91IGlkZWEhDQp3aGVuIGkgZm91bmQgYWZ0ZXIgaGl2ZS0w LjEzLHVzZXIgY2FuIGRlZmluZSBwZXJtYW5lbnQgVURGLmJ1dCBpdCBtdXN0IGJpbmQgd2l0aCBk YXRhYmFzZSBuYW1lLg0Kc28gaWYgd2Ugd2FudCB0byB1c2UgdGhlIHVkZiB3aXRob3V0IGRhdGFi YXNlIG5hbWUsd2UgbXVzdCBjcmVhdGUgaXQgYXQgYWxsIG9mIHRoZSBkYXRhYmFzZXMgbmFtZS4N Cml0IHRha2UgYW5vdGhlciBwcm9ibGVtLHdoZW4gd2UgY3JlYXRlIGEgbmV3IGRhdGFiYXNlcy53 ZSBuZWVkIGdldCBhbGwgb2YgdGhlIHVkZnMgdGhhdCB3ZSBoYXZlIGJlZW4gZGVmaW5lZC4NCnRo ZW4gY3JlYXRlIHRoZW0gb25lIGJ5IG9uZS4NClRoaXMgaXMgdGhlIGJpZ2dlc3QgcHJvYmxlbSBJ IGhhdmUgZW5jb3VudGVyZWQgaW4gdGhlIHVzZSBvZi4NCg0KamlwZW5nemVuZw0KDQoNCg0KIA0K RnJvbTogRnVyY3kgUGluDQpEYXRlOiAyMDE1LTEyLTE3IDIwOjE0DQpUbzogdXNlcg0KU3ViamVj dDogRGlzY3Vzc2lvbjogcGVybWFuZW50IFVERiB3aXRoIGRhdGFiYXNlIG5hbWUNCkhpIEhpdmUg dXNlcnMsDQoNCkkgd291bGQgbGlrZSB0byBwdXJzdWUgdGhlIGRpc2N1c3Npb24gdGhhdCBoYXBw ZW5lZCBkdXJpbmcgdGhlIGRlc2lnbiBvZiB0aGUgZmVhdHVyZToNCmh0dHBzOi8vaXNzdWVzLmFw YWNoZS5vcmcvamlyYS9icm93c2UvSElWRS02MTY3DQoNClNvbWUgY29uY2VybiB3aGVyZSByYWlz ZWQgYmFjayB0aGVuLCBhbmQgSSB0aGluayB0aGF0IG1heWJlIG5vdyB0aGF0IGl0IGhhcyBiZWVu IGltcGxlbWVudGVkLCBzb21lIHVzZXIgZmVlZGJhY2tzIGNvdWxkIGJyaW5nIHdhdGVyIHRvIHRo ZSBtaWxsLg0KDQpFdmVuIGlmIEkgdW5kZXJzdGFuZCB0aGUgdXRpbGl0eSBvZiBncm91cGluZyBV REZzIGluc2lkZSBkYXRhYmFzZXMsIEkgZmluZCBpdCByZWFsbHkgYW5ub3lpbmcgbm90IHRvIGJl IGFibGUgdG8gZGVmaW5lIG15IFVERnMgZ2xvYmFsbHkuDQoNCkZvciBtZSwgb25lIG9mIHRoZSBt YWluIGludGVyZXN0cyBvZiBVREZzIGlzIHRvIGV4dGVuZCB0aGUgYnVpbHQtaW4gSGl2ZSBmdW5j dGlvbnMgd2l0aCB0aGUgY29tcGFueSdzIHVzZXItZGVmaW5lZCBmdW5jdGlvbnMsIGVpdGhlciBi ZWNhdXNlIHNvbWUgdXNlZnVsIGdlbmVyaWMgZnVuY3Rpb24gYXJlIG1pc3NpbmcgaW4gdGhlIGJ1 aWx0LWluIGZ1bmN0aW9ucyBvciB0byBhZGQgYnVzaW5lc3Mtc3BlY2lmaWMgZnVuY3Rpb25zLg0K DQpJbiB0aGUgbGF0dGVyIGNhc2UsIEkgdW5kZXJzdGFuZCB2ZXJ5IHdlbGwgdGhlIG5lY2Vzc2l0 eSBvZiBxdWFsaWZ5aW5nIHRoZW0gd2l0aCBhIGJ1c2luZXNzLXNwZWNpZmljIGRhdGFiYXNlIG5h bWUuIEJ1dCBpbiB0aGUgZm9ybWVyIGNhc2U/DQoNCg0KTGV0J3MgdGFrZSBhbiBleGFtcGxlOg0K SXQgaGFwcGVuZWQgc2V2ZXJhbCB0aW1lcyB0aGF0IHdlIG5lZWRlZCBhIEhpdmUgVURGIHRoYXQg d2FzIGRpZCBub3QgZXhpc3QgeWV0IG9uIHRoZSBIaXZlIHZlcnNpb24gdGhhdCB3ZSB3ZXJlIGN1 cnJlbnRseSBydW5uaW5nLiBUbyB1c2UgaXQsIGFsbCB3ZSBoYWQgdG8gZG8gd2FzIHRha2UgdGhl IFVERidzIHNvdXJjZSBjb2RlIGZyb20gYSBtb3JlIHJlY2VudCB2ZXJzaW9uIG9mIEhpdmUsIGJ1 aWx0IGl0IGluIGEgSkFSLCBhbmQgYWRkIHRoZSBVREYgbWFudWFsbHkuDQoNCldoZW4gd2UgdXBn cmFkZWQsIHdlIG9ubHkgYWRkIHRvIHJlbW92ZSBvdXIgVURGIHNpbmNlIGl0IHdhcyBub3cgYnVp bHQtaW4uDQoNCihUbyBiZSBtb3JlIHNwZWNpZmljIGl0IGhhcHBlbmVkIHdpdGggY29sbGVjdF9s aXN0IHByaW9yIHRvIEhpdmUgMC4xMykuDQoNCldpdGggSElWRS02MTY3LCB0aGlzIGJlY2FtZSBp bXBvc3NpYmxlLCBzaW5jZSB3ZSBvdWdodCB0byBjcmVhdGUgYSAiZGF0YWJhc2VfbmFtZS5mdW5j dGlvbl9uYW1lIiwgYW5kIHVzZSBpdCBhcyBpcy4gSGVuY2UsIHdoZW4gdXBncmFkaW5nIHdlIG5l ZWQgdG8gcmVuYW1lIGV2ZXJ5d2hlcmUgImRhdGFiYXNlX25hbWUuZnVuY3Rpb25fbmFtZSIgd2l0 aCAiZnVuY3Rpb25fbmFtZSIuDQoNClRoaXMgaXMganVzdCBhbiBleGFtcGxlLCBidXQgSSB3b3Vs ZCBsaWtlIHRvIGVtcGhhc2l6ZSB0aGUgcG9pbnQgdGhhdCBzb21ldGltZXMgd2Ugd2FudCB0byBj cmVhdGUgcGVybWFuZW50IFVERnMgdGhhdCBhcmUgYXMgZ2xvYmFsIGFzIGJ1aWx0LWluIFVERnMg YW5kIG5vdCBib3RoZXIgaWYgaXQgaXMgYSBidWlsdC1pbiBvciB1c2VyLWRlZmluZWQgZnVuY3Rp b24uIEFzIHNvbWVvbmUgcG9pbnRlZCBvdXQgaW4gSElWRS02MTY3J3MgZGlzY3Vzc2lvbiwgaW1h Z2luZSBpZiBhbGwgdGhlIGJ1aWx0LWluIFVERnMgaGFkIHRvIGJlIGNhbGxlZCB3aXRoICJzeXMu ZnVuY3Rpb25fbmFtZSIuDQoNCkkgd291bGQganVzdCBsaWtlIHRvIGhhdmUgb3RoZXIgSGl2ZSB1 c2VyJ3MgZmVlZGJhY2sgb24gdGhhdCBtYXR0ZXIuDQoNCkRpZCBhbnlvbmUgZWxzZSBoYWQgc2lt aWxhciBpc3N1ZXMgd2l0aCB0aGlzIGJlaGF2aW9yPyBIb3cgZGlkIHlvdSB0cmVhdCB0aGVtPw0K DQpNYXliZSBpdCB3b3VsZCBtYWtlIHNlbnNlIHRvIGNyZWF0ZSBhIGZlYXR1cmUgcmVxdWVzdCBm b3IgYmVpbmcgYWJsZSB0byBzcGVjaWZ5IGEgR0xPQkFMIGtleXdvcmQgd2hlbiBjcmVhdGluZyBh IHBlcm1hbmVudCBVREYsIHdoZW4gd2UgcmVhbGx5IHdhbnQgaXQgdG8gYmUgZ2xvYmFsPw0KDQpX aGF0IGRvIHlvdSB0aGluaz8NCg0KUmVnYXJkcywNCg0KRnVyY3kNCg0K ------=_001_NextPart136843422704_=---- Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0A
= @ Furcy Pin
I agree you idea!
when i foun= d after hive-0.13,user can define  UDF.but it mu= st bind with database name.
so if we want to use the udf without database name,we must create = it at all of the databases name.
it take another problem,when we create a new databases.we nee= d get all of the udfs that we have been defined.
then create them one by one.
This is the biggest problem I&= nbsp;have encountered in the use of.=

=0A
jipengz= eng
=0A

=0A
 <= /div>
From: Furcy Pin
Date: 2015-12-17&= nbsp;20:14
To: user
Subject: Discussion: permanent UDF with = database name
Hi Hive users,

I wo= uld like to pursue the discussion that happened during the design of the f= eature:

Some concern where raised back then, and I think that maybe now tha= t it has been implemented, some user feedbacks could bring water to the mi= ll.

Even if I understand the utility of grouping = UDFs inside databases, I find it really annoying not to be able to define = my UDFs globally.

For me, one of the main interes= ts of UDFs is to extend the built-in Hive functions with the company's use= r-defined functions, either because some useful generic function are missi= ng in the built-in functions or to add business-specific functions.
<= div>
In the latter case, I understand very well the necessit= y of qualifying them with a business-specific database name. But in the fo= rmer case?


Let's take an example:<= /div>
It happened several times that we needed a Hive UDF that was did= not exist yet on the Hive version that we were currently running. To use = it, all we had to do was take the UDF's source code from a more recent ver= sion of Hive, built it in a JAR, and add the UDF manually.

<= /div>
When we upgraded, we only add to remove our UDF since it was now= built-in.

(To be more specific it happened with = collect_list prior to Hive 0.13).

With HIVE-6167,= this became impossible, since we ought to create a "database_name.functio= n_name", and use it as is. Hence, when upgrading we need to rename everywh= ere "database_name.function_name" with "function_name".

This is just an example, but I would like to emphasize the point th= at sometimes we want to create permanent UDFs that are as global as built-= in UDFs and not bother if it is a built-in or user-defined function. As so= meone pointed out in HIVE-6167's discussion, imagine if all the built-in U= DFs had to be called with "sys.function_name".

I = would just like to have other Hive user's feedback on that matter.

Did anyone else had similar issues with this behavior? H= ow did you treat them?

Maybe it would make sense = to create a feature request for being able to specify a GLOBAL keyword whe= n creating a permanent UDF, when we really want it to be global?

What do you think?

Regards,
=

Furcy

=0A
=0A ------=_001_NextPart136843422704_=------