Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of remusr@microsoft.com designates
 207.46.163.206 as permitted sender)
From: Remus Rusanu <remusr@microsoft.com>
To: "dev@hive.apache.org" <dev@hive.apache.org>
CC: "xuefu@apache.org" <xuefu@apache.org>, "Eric Hanson (BIG DATA)"
	<Eric.N.Hanson@microsoft.com>
Subject: RE: A question about the derivation of intermediate sum field for
 decimal average aggregates
Thread-Topic: A question about the derivation of intermediate sum field for
 decimal average aggregates
Thread-Index: Ac8peiutU7iy0/YYQJ+StopTkuk+gAALpFcAAAoHvhA=
Date: Fri, 14 Feb 2014 22:09:02 +0000
Message-ID: 
 <44e894c2083d4545ab5dd2989f099349@BY2PR03MB412.namprd03.prod.outlook.com>
References: 
 <26098f415a4e419cb002d6f1fa7a44e7@BY2PR03MB412.namprd03.prod.outlook.com>
 <CADx-ob3NquKZ=RjeNnHTZ_XFDOV2=JfMFGLdWipTbmN=yTKLwQ@mail.gmail.com>
In-Reply-To: 
 <CADx-ob3NquKZ=RjeNnHTZ_XFDOV2=JfMFGLdWipTbmN=yTKLwQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Xuefu,

I do not have any particular use case in mind. I've noticed the problem whe=
n I implemented the vectorized AVG for decimal, which must match your imple=
mentation (since we vectorized only the map side operator, it better produc=
e the output expected by the reduce side...).  I thought that since we alte=
r the precission/scale for the result, we may as well alter it for the inte=
rmediate sum field. But if this complicates the use of object inspectors an=
d introduces maintenance risks, then is probably not worth it.

Thanks,
~Remus

-----Original Message-----
From: Xuefu Zhang [mailto:xzhang@cloudera.com]=20
Sent: Friday, February 14, 2014 7:18 PM
To: dev@hive.apache.org
Cc: xuefu@apache.org; Eric Hanson (BIG DATA)
Subject: Re: A question about the derivation of intermediate sum field for =
decimal average aggregates

Remus,

Thanks for looking into this. You're right that sum() result doesn't increa=
se the scale, but have you seen that sum UDF returns wrong scale?

As to the implementation of avg UDF, the object inspector for sum field is =
initialized with a scale + 4, which might not be necessary, but perhaps har=
mless. The same object inspector is also used for the average result, which=
 gives correct type. I guess it's possible to separate this into two object=
 inspectors, one for sum field and one for the avg result, but the differen=
ce might be subtle and questionable. This is because the data may not compl=
y to the metadata specified for Hive tables. Thus, I'm not sure if truncati=
ng data before it's summed if the right behavior.

Do you have a use case that suggests one is better than the other?

--Xuefu


On Fri, Feb 14, 2014 at 3:55 AM, Remus Rusanu <remusr@microsoft.com> wrote:

> Hi,
>
> With HIVE-5872 the intermediate sum field for decimal aggregates was=20
> changed to increase scale by 4. I understand the reasoning for having=20
> accurate precision/scale for the aggregate output. However, for the=20
> intermediate sum field of AVG, I believe we should increase precision=20
> w/o increasing scale. The sum can grow large, but cannot increase=20
> digits in the fractional part, so we should increase the precision of=20
> the sum, but not the scale. When sum is divided by count to get the=20
> average on the reduce side then we should indeed project value with highe=
r scale.
>
> Opinions?
>
> Thanks,
> ~Remus
>
>