Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of john.meagher@gmail.com
 designates 209.85.215.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKOFcwpmAok=zq4xQxZB1YNi5P+mEY0ZekE2=XnxWGngyyPnaw@mail.gmail.com>
References: 
 <CAKOFcwpmAok=zq4xQxZB1YNi5P+mEY0ZekE2=XnxWGngyyPnaw@mail.gmail.com>
Date: Tue, 19 Feb 2013 10:03:42 -0500
Message-ID: 
 <CAKb7DdhUEZseOV=0NZ8vN1-h9kVLGZX4dqTiWn9JPSRBCm7VGA@mail.gmail.com>
Subject: Re: Using Reflect: A thread for ideas
From: John Meagher <john.meagher@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=f46d04095bf7bbe7be04d615254d

--f46d04095bf7bbe7be04d615254d
Content-Type: text/plain; charset=ISO-8859-1

Another option for this functionality would be to use the Java scripting
API.  The basic structure of the call would be...

select script( scriptLanguage, scriptToRun, args... )

I haven't seen that in Hive, but something similar is available for Pig.
 Documentation for that is available on
http://pig.apache.org/docs/r0.9.2/udf.html#js-udfs.  There's also a
variation in Jira https://issues.apache.org/jira/browse/PIG-1777.


On Wed, Feb 13, 2013 at 11:38 PM, John Omernik <john@omernik.com> wrote:

> I stumbled across the little documented reflect function today. I've
> always known about it, but java scares me if it's not in a cup so I didn't
> dig.  Well today I dug, and found an awesome use case for reflect (for me)
> and wanted to share.  I also thought it would be nice to validate some
> thoughts I had on reflect, and how we could possibly share ideas on reflect
> so that folks could get more use out of this great feature of hive.
>
> Here's my example: A simple URL decode function:
>
> select url, reflect('java.net.URLDecoder', 'decode', url, 'utf-8') as
> decoded_url from logs
> Basically I am using the decode function of the java.net.URLDecoder class.
>  Pretty awesome, works great, no files to distribute either.  Even works
> through JDBC!
>
> Ok that being said, I realized now that the function I am trying to call
> has to return data in a simple data type.  For example, I struggle to come
> up with a simple reflect() for making an Hex MD5 out of a string because
> the built in function return an object, which have methods that can return
> what I am looking for. Which is great, but then I have to compile java
> code, distribute a jar, and then run the code. I am looking for simple like
> the URLDecoding function.
>
> I love this reflect feature, but I think it's probably underutilized due
> to the perceived usability issues for beginners.  So that leads me to my
> next thought. What if we brain storm here handy functions in Java that are
> not included in the standard hive language, that make the transition to
> hive well using the reflect function and the show an example of it's use? I
> went first with my URLDecode, and obviously will be looking for more, but
> have you seen some examples that we neat and worked well for you? Can you
> share?
>
> Perhaps if we get enough examples we could roll some of these into a wiki
> page on the hive wiki that folks can use to get over the "perceived"
> complexity of using java reflect?
>
> Thanks to those who have worked hard to implement features like this, it
> is truly awesome.
>

--f46d04095bf7bbe7be04d615254d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Another option for this functionality would be to use the =
Java scripting API. =A0The basic structure of the call would be...<div><br>=
</div><div style>select script( scriptLanguage, scriptToRun, args... )</div=
>
<div style><br></div><div style>I haven&#39;t seen that in Hive, but someth=
ing similar is available for Pig. =A0Documentation for that is available on=
=A0<a href=3D"http://pig.apache.org/docs/r0.9.2/udf.html#js-udfs">http://pi=
g.apache.org/docs/r0.9.2/udf.html#js-udfs</a>. =A0There&#39;s also a variat=
ion in Jira=A0<a href=3D"https://issues.apache.org/jira/browse/PIG-1777">ht=
tps://issues.apache.org/jira/browse/PIG-1777</a>.</div>
<div style><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D=
"gmail_quote">On Wed, Feb 13, 2013 at 11:38 PM, John Omernik <span dir=3D"l=
tr">&lt;<a href=3D"mailto:john@omernik.com" target=3D"_blank">john@omernik.=
com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">I stumbled across the little documented refl=
ect function today. I&#39;ve always known about it, but java scares me if i=
t&#39;s not in a cup so I didn&#39;t dig. =A0Well today I dug, and found an=
 awesome use case for reflect (for me) and wanted to share. =A0I also thoug=
ht it would be nice to validate some thoughts I had on reflect, and how we =
could possibly share ideas on reflect so that folks could get more use out =
of this great feature of hive.=A0<div>


<br></div><div>Here&#39;s my example: A simple URL decode function:</div><d=
iv><br></div><div>select url,=A0reflect(&#39;java.net.URLDecoder&#39;, &#39=
;decode&#39;, url, &#39;utf-8&#39;) as decoded_url from logs</div><div>Basi=
cally I am using the decode function of the java.net.URLDecoder class. =A0P=
retty awesome, works great, no files to distribute either. =A0Even works th=
rough JDBC!</div>


<div><br></div><div>Ok that being said, I realized now that the function I =
am trying to call has to return data in a simple data type. =A0For example,=
 I struggle to come up with a simple reflect() for making an Hex MD5 out of=
 a string because the built in function return an object, which have method=
s that can return what I am looking for. Which is great, but then I have to=
 compile java code, distribute a jar, and then run the code. I am looking f=
or simple like the URLDecoding function. =A0</div>


<div><br></div><div>I love this reflect feature, but I think it&#39;s proba=
bly underutilized due to the perceived usability issues for beginners. =A0S=
o that leads me to my next thought. What if we brain storm here handy funct=
ions in Java that are not included in the standard hive language, that make=
 the transition to hive well using the reflect function and the show an exa=
mple of it&#39;s use? I went first with my URLDecode, and obviously will be=
 looking for more, but have you seen some examples that we neat and worked =
well for you? Can you share? =A0</div>


<div><br></div><div>Perhaps if we get enough examples we could roll some of=
 these into a wiki page on the hive wiki that folks can use to get over the=
 &quot;perceived&quot; complexity of using java reflect? =A0</div><div><br>


</div><div>Thanks to those who have worked hard to implement features like =
this, it is truly awesome.=A0</div>
</blockquote></div><br></div>

--f46d04095bf7bbe7be04d615254d--