hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-14204) Optimize loading dynamic partitions
Date Thu, 21 Jul 2016 10:15:20 GMT

     [ https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated HIVE-14204:
------------------------------------
    Attachment: HIVE-14204.4.patch

sync access to MSC would impact perf when compared with initial improvements. But still the
diff compared without the patch is significant. Attaching the patch which uses SynchronizedMetaStoreClient().
Moved this class outside of DbTxnManager. 

Without Patch:
==============
INSERT INTO TABLE  web_sales_test partition(ws_sold_date_sk) select * from tpcds_bin_partitioned_orc_200.web_sales;
Time taken to load dynamic partitions: *354.176 seconds*

With Patch (004):
================
INSERT INTO TABLE  web_sales_test partition(ws_sold_date_sk) select * from tpcds_bin_partitioned_orc_200.web_sales;
Time taken to load dynamic partitions: *122.517 seconds*



> Optimize loading dynamic partitions 
> ------------------------------------
>
>                 Key: HIVE-14204
>                 URL: https://issues.apache.org/jira/browse/HIVE-14204
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HIVE-14204.1.patch, HIVE-14204.3.patch, HIVE-14204.4.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned dataset in driver
side. E.g simple dynamic partitioned load as follows takes 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message