hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Meagher <john.meag...@gmail.com>
Subject Re: Can data be passed to the final mode init call in a UDAF?
Date Wed, 12 Feb 2014 14:41:14 GMT
The code I'm having problems with is
https://github.com/livingsocial/HiveSwarm/blob/WIP_scripting/src/main/java/com/livingsocial/hive/udf/ScriptedUDAF.java.

There is a FIXME comment at the problem area where the return type for
the init method can not be determined.

On Mon, Feb 10, 2014 at 4:26 PM, John Meagher <john.meagher@gmail.com> wrote:
> I'm working on a UDAF that takes in a constant string that defines
> what the final output of the UDAF will be.  In the mode=PARTIAL1 call
> to the init function all the parameters are available and the constant
> can be read so the output ObjectInspector can be built.  I haven't
> found a way to pass this same constant data through so it is available
> to the mode=FINAL call to init.
>
> Is there a way to pass constant parameters through the UDAF so it is
> available in the mode=FINAL init call?
>
> I have tried having the PARTIAL1 init call return a
> StructObjectInspector containing constant object inspectors with the
> data, but when that is read during the FINAL call the incoming
> ObjectInspectors are no longer constants.
>
> I've looked for ways to access the JobConf object for the job to try
> to pass this parameter along, but I can't find a way to access that
> from Hive.  I can pass the data along with the data coming from
> terminatePartial, but that data is not available until merge is called
> and init in FINAL mode may be called without another invocation of
> init.
>
> Note: this is with Hive 0.10.  While I'd like to get this working
> under 0.10 I'd like to know if this is a feature available in newer
> releases.
>
> Thanks,
> John

Mime
View raw message