hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13306) Better Decimal vectorization
Date Tue, 06 Dec 2016 15:55:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725882#comment-15725882
] 

Hive QA commented on HIVE-13306:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12841966/HIVE-13306.4.patch

{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10785 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_17] (batchId=79)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf] (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_17] (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1] (batchId=91)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress] (batchId=113)
org.apache.hadoop.hive.ql.exec.vector.TestDecimalColumnVector.testA (batchId=167)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2442/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2442/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2442/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12841966 - PreCommit-HIVE-Build

> Better Decimal vectorization
> ----------------------------
>
>                 Key: HIVE-13306
>                 URL: https://issues.apache.org/jira/browse/HIVE-13306
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>            Reporter: Matt McCline
>            Assignee: Teddy Choi
>            Priority: Critical
>         Attachments: HIVE-13306.1.patch, HIVE-13306.2.patch, HIVE-13306.3.patch, HIVE-13306.4.patch
>
>
> Decimal Vectorization Requirements
> •	Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector, TimestampColumnVector
classes store the data as primitive Java data types long, double, or byte arrays for efficiency.
> •	DecimalColumnVector is different - it has an array of Object references to HiveDecimal
objects.
> •	The HiveDecimal object uses an internal object BigDecimal for its implementation.
 Further, BigDecimal itself uses an internal object BigInteger for its implementation, and
BigInteger uses an int array.  4 objects total.
> •	And, HiveDecimal is an immutable object which means arithmetic and other operations
produce new HiveDecimal object with 3 new objects underneath.
> •	A major reason Vectorization is fast is the ColumnVector classes except DecimalColumnVector
do not have to allocate additional memory per row.   This avoids memory fragmentation and
pressure on the Java Garbage Collector that DecimalColumnVector can generate.  It is very
significant.
> •	What can be done with DecimalColumnVector to make it much more efficient?
> o	Design several new decimal classes that allow the caller to manage the decimal storage.
> o	If it takes N int values to store a decimal (e.g. N=1..5), then a new DecimalColumnVector
would have an int[] of length N*1024 (where 1024 is the default column vector size).
> o	Why store a decimal in separate int values?
> •	Java does not support 128 bit integers.
> •	Java does not support unsigned integers.
> •	In order to do multiplication of a decimal represented in a long you need twice the
storage (i.e. 128 bits).  So you need to represent parts in 32 bit integers.
> •	But really since we do not have unsigned, really you can only do multiplications
on N-1 bits or 31 bits.
> •	So, 5 ints are needed for decimal storage... of 38 digits.
> o	It makes sense to have just one algorithm for decimals rather than one for HiveDecimal
and another for DecimalColumnVector.  So, make HiveDecimal store N int values, too.
> o	A lower level primitive decimal class would accept decimals stored as int arrays and
produces results into int arrays.  It would be used by HiveDecimal and DecimalColumnVector.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message