hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Balamohan <>
Subject Re: Review Request 49881: HIVE-14204: Optimize loading loaddynamic partitions
Date Wed, 03 Aug 2016 07:49:42 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 3, 2016, 7:49 a.m.)

Review request for hive and Ashutosh Chauhan.


- Removed fetching existing partitions in loadDynamicPartitions. This can be added as a follow
on optimization later.

Bugs: HIVE-14204

Repository: hive-git


Lots of time is spent in sequential fashion to load dynamic partitioned dataset in driver

E.g simple dynamic partitioned load as follows takes 300+ seconds

INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from tpcds_bin_partitioned_orc_200.web_sales;

Time taken to load dynamic partitions: 309.22 seconds

Diffs (updated)

  common/src/java/org/apache/hadoop/hive/conf/ aa7647b 
  metastore/src/java/org/apache/hadoop/hive/metastore/ 5adfa02 
  metastore/src/java/org/apache/hadoop/hive/metastore/ d624d1b 
  ql/src/java/org/apache/hadoop/hive/metastore/ PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/ b4ae1d1 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/ 02c17b5 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/ 9d927bd 




Rajesh Balamohan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message