Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 6 Oct 2016 05:37:20 +0000 (UTC)
From: "Teddy Choi (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.12951386.1458258434000.750062.1475732240500@Atlassian.JIRA>
In-Reply-To: <JIRA.12951386.1458258434000@Atlassian.JIRA>
References: <JIRA.12951386.1458258434000@Atlassian.JIRA> <JIRA.12951386.1458258434645@arcas>
Subject: [jira] [Updated] (HIVE-13306) Better Decimal vectorization
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 06 Oct 2016 05:37:22 -0000


     [ https://issues.apache.org/jira/browse/HIVE-13306?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

Teddy Choi updated HIVE-13306:
------------------------------
    Attachment: HIVE-13306.3.patch

Implemented with long arrays

> Better Decimal vectorization
> ----------------------------
>
>                 Key: HIVE-13306
>                 URL: https://issues.apache.org/jira/browse/HIVE-13306
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>            Reporter: Matt McCline
>            Assignee: Teddy Choi
>            Priority: Critical
>         Attachments: HIVE-13306.1.patch, HIVE-13306.2.patch, HIVE-13306.3=
.patch
>
>
> Decimal Vectorization Requirements
> =E2=80=A2=09Today, the LongColumnVector, DoubleColumnVector, BytesColumnV=
ector, TimestampColumnVector classes store the data as primitive Java data =
types long, double, or byte arrays for efficiency.
> =E2=80=A2=09DecimalColumnVector is different - it has an array of Object =
references to HiveDecimal objects.
> =E2=80=A2=09The HiveDecimal object uses an internal object BigDecimal for=
 its implementation.  Further, BigDecimal itself uses an internal object Bi=
gInteger for its implementation, and BigInteger uses an int array.  4 objec=
ts total.
> =E2=80=A2=09And, HiveDecimal is an immutable object which means arithmeti=
c and other operations produce new HiveDecimal object with 3 new objects un=
derneath.
> =E2=80=A2=09A major reason Vectorization is fast is the ColumnVector clas=
ses except DecimalColumnVector do not have to allocate additional memory pe=
r row.   This avoids memory fragmentation and pressure on the Java Garbage =
Collector that DecimalColumnVector can generate.  It is very significant.
> =E2=80=A2=09What can be done with DecimalColumnVector to make it much mor=
e efficient?
> o=09Design several new decimal classes that allow the caller to manage th=
e decimal storage.
> o=09If it takes N int values to store a decimal (e.g. N=3D1..5), then a n=
ew DecimalColumnVector would have an int[] of length N*1024 (where 1024 is =
the default column vector size).
> o=09Why store a decimal in separate int values?
> =E2=80=A2=09Java does not support 128 bit integers.
> =E2=80=A2=09Java does not support unsigned integers.
> =E2=80=A2=09In order to do multiplication of a decimal represented in a l=
ong you need twice the storage (i.e. 128 bits).  So you need to represent p=
arts in 32 bit integers.
> =E2=80=A2=09But really since we do not have unsigned, really you can only=
 do multiplications on N-1 bits or 31 bits.
> =E2=80=A2=09So, 5 ints are needed for decimal storage... of 38 digits.
> o=09It makes sense to have just one algorithm for decimals rather than on=
e for HiveDecimal and another for DecimalColumnVector.  So, make HiveDecima=
l store N int values, too.
> o=09A lower level primitive decimal class would accept decimals stored as=
 int arrays and produces results into int arrays.  It would be used by Hive=
Decimal and DecimalColumnVector.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)