hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Ma (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
Date Tue, 28 Mar 2017 07:33:41 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944685#comment-15944685
] 

Colin Ma edited comment on HIVE-16311 at 3/28/17 7:33 AM:
----------------------------------------------------------

The initial patch is uploaded.
Do the simple test, 12345.67/123.45  with FastHiveDecimalImpl.fastDivide 500000 times, the
result shows 1s(without patch) vs  0.1s(with patch).
Also test the patch with q06 of TPCx-BB which has the following divide expression:  
{code}
sum( case when (d_year = 2001)   THEN (((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2)
  ELSE 0 END) first_year_total
{code}
The cluster includes 6 nodes, 128G memory/per node, CPU is Intel(R) Xeon(R) E5-2680, 1G network.
With the 1T data scale and spark as executor engine, the following is the result:
|| ||without patch||with patch||improvement||
|disable vectorization|214s|164s|23.36%|
|enable vectorization(Parquet file format)|252s|125s|50.4%|


was (Author: colinma):
The initial patch is uploaded.
Do the simple test, 12345.67/123.45  with FastHiveDecimalImpl.fastDivide 500000 times, the
result shows 1s(without patch) vs  0.1s(with patch).
Also test the patch with q06 of TPCx-BB which has the following divide expression:  
{code}
sum( case when (d_year = 2001)   THEN (((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2)
  ELSE 0 END) first_year_total
{code}
The cluster includes 6 nodes, 128G memory/per node, CPU is Intel(R) Xeon(R) E5-2680, 1G network.
With the 1T data scale and spark as executor engine, the following is the result:
|| ||without patch||with patch||improvement||
|disable vectorization|214s|164s|23.36%|
|enable vectorization|252s|125s|50.4%|

> Improve the performance for FastHiveDecimalImpl.fastDivide
> ----------------------------------------------------------
>
>                 Key: HIVE-16311
>                 URL: https://issues.apache.org/jira/browse/HIVE-16311
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 2.2.0
>            Reporter: Colin Ma
>            Assignee: Colin Ma
>             Fix For: 2.2.0
>
>         Attachments: HIVE-16311.001.patch
>
>
> FastHiveDecimalImpl.fastDivide is poor performance when evaluate the expression as 12345.67/123.45
> There are 2 points can be improved:
> 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the BigDecimal.divide.
> 2. Get the precision for BigInteger in a fast way if possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message