hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Connell, Chuck" <Chuck.Conn...@nuance.com>
Subject RE: CHAN (Comprehensive Hive Archive Network) (Was Re: Using Reflect: A thread for ideas)
Date Thu, 14 Feb 2013 13:58:33 GMT
+1, great idea!

Chuck Connell, Nuance

From: Robin Morris [mailto:rdm@baynote.com]
Sent: Thursday, February 14, 2013 1:59 AM
To: user@hive.apache.org
Subject: CHAN (Comprehensive Hive Archive Network) (Was Re: Using Reflect: A thread for ideas)

I think we need to think a little bigger than this.

Recently I've been thinking that what would be most useful to the Hive user community would
be a CHAN - Comprehensive Hive Archive Network, (analogous to CPAN, CRAN, CTAN etc.).  A central
place where user-contributed UD[A,T]Fs could be uploaded and made available to the wider community.

Is there interest in the user community for something like this?

Robin

From: John Omernik <john@omernik.com<mailto:john@omernik.com>>
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Date: Wednesday, February 13, 2013 8:38 PM
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Using Reflect: A thread for ideas

I stumbled across the little documented reflect function today. I've always known about it,
but java scares me if it's not in a cup so I didn't dig.  Well today I dug, and found an awesome
use case for reflect (for me) and wanted to share.  I also thought it would be nice to validate
some thoughts I had on reflect, and how we could possibly share ideas on reflect so that folks
could get more use out of this great feature of hive.

Here's my example: A simple URL decode function:

select url, reflect('java.net.URLDecoder', 'decode', url, 'utf-8') as decoded_url from logs
Basically I am using the decode function of the java.net.URLDecoder class.  Pretty awesome,
works great, no files to distribute either.  Even works through JDBC!

Ok that being said, I realized now that the function I am trying to call has to return data
in a simple data type.  For example, I struggle to come up with a simple reflect() for making
an Hex MD5 out of a string because the built in function return an object, which have methods
that can return what I am looking for. Which is great, but then I have to compile java code,
distribute a jar, and then run the code. I am looking for simple like the URLDecoding function.

I love this reflect feature, but I think it's probably underutilized due to the perceived
usability issues for beginners.  So that leads me to my next thought. What if we brain storm
here handy functions in Java that are not included in the standard hive language, that make
the transition to hive well using the reflect function and the show an example of it's use?
I went first with my URLDecode, and obviously will be looking for more, but have you seen
some examples that we neat and worked well for you? Can you share?

Perhaps if we get enough examples we could roll some of these into a wiki page on the hive
wiki that folks can use to get over the "perceived" complexity of using java reflect?

Thanks to those who have worked hard to implement features like this, it is truly awesome.

Mime
View raw message