Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
From: Aurora Skarra-Gallagher <aurora@yahoo-inc.com>
To: "user@hive.apache.org" <user@hive.apache.org>
CC: "Christopher, Pat" <patrick.christopher@hp.com>,
        Steven Wong
	<swong@netflix.com>
Date: Fri, 11 Mar 2011 19:00:12 -0800
Subject: Re: UDAF documentation
Thread-Topic: UDAF documentation
Thread-Index: AcvgYZtqsM29Y4wDTMKl+BjSMpd5kQ==
Message-ID: <7ABBE390-E832-462A-B743-DABF9694CA6F@yahoo-inc.com>
References: 
 <FB0065ADBE630748A6D362671F0DE4E725A937D2CC@GVW0432EXB.americas.hpqcorp.net>
 <AANLkTi=t=WD-6bO=0ra9n7CJ3r0S_uQGAFcoDhMucWrf@mail.gmail.com>
 <4F6B25AFFFCAFE44B6259A412D5F9B102DED87E0@ExchMBX104.netflix.com>
 <FB0065ADBE630748A6D362671F0DE4E725A93EEE34@GVW0432EXB.americas.hpqcorp.net>
 <E357C6E8-64BF-4D5A-BC77-759A9175AB11@yahoo-inc.com>
 <FB0065ADBE630748A6D362671F0DE4E725A93EF174@GVW0432EXB.americas.hpqcorp.net>
 <AF912632-1523-4821-BD82-CE354814BE37@yahoo-inc.com>
 <08A92376-1AE0-47D4-8C92-D6979FFD7214@yahoo-inc.com>
In-Reply-To: <08A92376-1AE0-47D4-8C92-D6979FFD7214@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I'll just keep responding to myself. ;)

I ended up figuring out how to do it. I just used junit and called init, it=
erate, terminatePartial, etc from inside the unit test. After knowing a typ=
ical flow of function calls (as I mentioned below), the main other gotcha i=
s making sure to have a new UDAF object for each instance. For example, in =
my example below, there would be three separate UDAF instances.

-Aurora

On Mar 11, 2011, at 5:02 PM, Aurora Skarra-Gallagher wrote:

> I'm looking for something like this, but for a UDAF instead of a UDF:
> http://svn.apache.org/repos/asf/hive/branches/branch-0.7/ql/src/test/org/=
apache/hadoop/hive/ql/udf/TestUDFDateDiff.java
>=20
> -Aurora
>=20
> On Mar 11, 2011, at 4:44 PM, Aurora Skarra-Gallagher wrote:
>=20
>> Hi,
>>=20
>> Did you actually call those functions directly from your unit tests? I'm=
 looking for examples of that working, but all I see reference to are tests=
 to make sure the query produces the expected output (rather than directly =
testing the UDAF).
>>=20
>> -Aurora
>>=20
>> On Mar 11, 2011, at 3:44 PM, Christopher, Pat wrote:
>>=20
>>> Awesome, awesome.  That's what I had pieced together from Steve and Ed'=
s emails.  Glad to get confirmation on it.
>>>=20
>>> Its also what I did for my unit testing.  I also called everything with=
 null arguments to make sure those got handled gracefully.
>>>=20
>>> Pat
>>>=20
>>> -----Original Message-----
>>> From: Aurora Skarra-Gallagher [mailto:aurora@yahoo-inc.com]=20
>>> Sent: Friday, March 11, 2011 3:40 PM
>>> To: user@hive.apache.org
>>> Cc: Steven Wong
>>> Subject: Re: UDAF documentation
>>>=20
>>> Hadoop: The Definitive Guide has a good section on this. Chapter 12: Hi=
ve: User Defined Functions. It has a diagram that shows how things are call=
ed and when. The example I'm looking at shows this sequence:
>>>=20
>>> (first instance)
>>> init()
>>> iterate(1)
>>> iterate(2)
>>> iterate(3)
>>> terminatePartial()
>>>=20
>>> (second instance)
>>> init()
>>> iterate(4)
>>> iterate(2)
>>> terminatePartial()
>>>=20
>>> (then)
>>> init()
>>> merge(3)
>>> merge(4)
>>> terminate()
>>>=20
>>> The UDAF being described is a max integer function, hence the merge end=
ing up with the highest integer from each instance.
>>>=20
>>> -Aurora
>>>=20
>>> On Mar 11, 2011, at 9:54 AM, Christopher, Pat wrote:
>>>=20
>>>> Ahh, perfect.  The docs don't agree terribly well but the case study i=
s great.  The context for when merge() gets called was not clear to me.
>>>>=20
>>>> Thanks guys!
>>>>=20
>>>> Pat
>>>>=20
>>>> -----Original Message-----
>>>> From: Steven Wong [mailto:swong@netflix.com]=20
>>>> Sent: Thursday, March 10, 2011 6:24 PM
>>>> To: user@hive.apache.org
>>>> Cc: Christopher, Pat
>>>> Subject: RE: UDAF documentation
>>>>=20
>>>> Take a look at http://wiki.apache.org/hadoop/Hive/GenericUDAFCaseStudy=
, in case you haven't found it already.
>>>>=20
>>>>=20
>>>> -----Original Message-----
>>>> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]=20
>>>> Sent: Thursday, March 10, 2011 6:18 PM
>>>> To: user@hive.apache.org
>>>> Cc: Christopher, Pat
>>>> Subject: Re: UDAF documentation
>>>>=20
>>>> On Thu, Mar 10, 2011 at 8:27 PM, Christopher, Pat
>>>> <patrick.christopher@hp.com> wrote:
>>>>> Hi Guys,
>>>>>=20
>>>>> I'm writing a UDAF to run against hive 0.5 or hive 0.7.  The document=
ation I
>>>>> can find says to implement UDAFEvaluator and ensure that you implemen=
t
>>>>> init() , aggregate() and evaluate().  However, all of the examples I =
can
>>>>> find implement init(), iterate(), merge(), terminatePartial() and
>>>>> terminate().
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> What's the difference and where I can find the documentation on how t=
o write
>>>>> a UDAF?
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> Thanks,
>>>>>=20
>>>>> Pat
>>>>=20
>>>> At time the documentation may lag behind the code. I would checkout
>>>> the hive source code for the version you are working with and base
>>>> your work on other already existing UDAF's that are similar.
>>>>=20
>>>> Edward
>>>>=20
>>>=20
>>=20
>=20