Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 17 Mar 2016 23:47:33 +0000 (UTC)
From: "Matt McCline (JIRA)" <jira@apache.org>
To: dev@hive.apache.org
Message-ID: <JIRA.12951386.1458258434000.54524.1458258453460@Atlassian.JIRA>
In-Reply-To: <JIRA.12951386.1458258434000@Atlassian.JIRA>
References: <JIRA.12951386.1458258434000@Atlassian.JIRA>
 <JIRA.12951386.1458258434645@arcas>
Subject: [jira] [Created] (HIVE-13306) Better Decimal vectorization
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Matt McCline created HIVE-13306:
-----------------------------------

             Summary: Better Decimal vectorization
                 Key: HIVE-13306
                 URL: https://issues.apache.org/jira/browse/HIVE-13306
             Project: Hive
          Issue Type: Bug
          Components: Hive
            Reporter: Matt McCline
            Priority: Critical


Decimal Vectorization Requirements

=E2=80=A2=09Today, the LongColumnVector, DoubleColumnVector, BytesColumnVec=
tor, TimestampColumnVector classes store the data as primitive Java data ty=
pes long, double, or byte arrays for efficiency.
=E2=80=A2=09DecimalColumnVector is different - it has an array of Object re=
ferences to HiveDecimal objects.
=E2=80=A2=09The HiveDecimal object uses an internal object BigDecimal for i=
ts implementation.  Further, BigDecimal itself uses an internal object BigI=
nteger for its implementation, and BigInteger uses an int array.  4 objects=
 total.
=E2=80=A2=09And, HiveDecimal is an immutable object which means arithmetic =
and other operations produce new HiveDecimal object with 3 new objects unde=
rneath.
=E2=80=A2=09A major reason Vectorization is fast is the ColumnVector classe=
s except DecimalColumnVector do not have to allocate additional memory per =
row.   This avoids memory fragmentation and pressure on the Java Garbage Co=
llector that DecimalColumnVector can generate.  It is very significant.
=E2=80=A2=09What can be done with DecimalColumnVector to make it much more =
efficient?
o=09Design several new decimal classes that allow the caller to manage the =
decimal storage.
o=09If it takes N int values to store a decimal (e.g. N=3D1..5), then a new=
 DecimalColumnVector would have an int[] of length N*1024 (where 1024 is th=
e default column vector size).
o=09Why store a decimal in separate int values?
=E2=80=A2=09Java does not support 128 bit integers.
=E2=80=A2=09Java does not support unsigned integers.
=E2=80=A2=09In order to do multiplication of a decimal represented in a lon=
g you need twice the storage (i.e. 128 bits).  So you need to represent par=
ts in 32 bit integers.
=E2=80=A2=09But really since we do not have unsigned, really you can only d=
o multiplications on N-1 bits or 31 bits.
=E2=80=A2=09So, 5 ints are needed for decimal storage... of 38 digits.
o=09It makes sense to have just one algorithm for decimals rather than one =
for HiveDecimal and another for DecimalColumnVector.  So, make HiveDecimal =
store N int values, too.
o=09A lower level primitive decimal class would accept decimals stored as i=
nt arrays and produces results into int arrays.  It would be used by HiveDe=
cimal and DecimalColumnVector.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)