hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Pena <sergio.p...@cloudera.com>
Subject Re: Review Request 54393: HIVE-15361: INSERT dynamic partition on S3 fails with a MoveTask failure
Date Tue, 06 Dec 2016 20:13:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54393/
-----------------------------------------------------------

(Updated Dec. 6, 2016, 8:13 p.m.)


Review request for hive.


Changes
-------

Addressed all Illya comments.


Bugs: HIVE-15361
    https://issues.apache.org/jira/browse/HIVE-15361


Repository: hive-git


Description
-------

Problem:
- DynamicPartitionCtx and ListBucketingCtx objects weren't set on the new MoveWork created
when merging the two MoveWork objects from the ConditionalTask.

Solution
- Set the DynamicPartitionCtx and ListBucketingCtx objects to the new MoveWork created for
the S3 optimization.

Other changes
- Merge the MoveWork objects inside the createCondTask() method for better error handling.
- Only merge the MoveWork related to the moveOnlyMoveTask. The MoveWork from the mergeAndMoveMoveTask
may cause other issues that are not correctly tested.
- Two new private methods are added to check and merge the conditional input/output tasks
to the linked MoveWork.


Diffs (updated)
-----

  itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q PRE-CREATION

  itests/hive-blobstore/src/test/queries/clientpositive/insert_into_table.q 25e2e7007ff539223d9244ca9822aa65d1441eb0

  itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions.q
PRE-CREATION 
  itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_table.q 846b2b113f09a74a3f05c13ffb56163e81dc1e8e

  itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
PRE-CREATION 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out fbb52c132a331aefe870264e035c397078f3c82e

  itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out 9f575a66ecefc3933b16dff554bdcc1c1f6420ee

  itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
PRE-CREATION 
  itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out c725c96cbb6b0374e67308a54204c7c25e827567

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java adc1188f09c8019a8aa60403d5813d6fa4509ceb

  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadDesc.java bcd3125ab4ad20c00fec565e5004ee200c0187d5

  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 9a868a04ce93d5c2ee75b5c6e96a1401cea93133

  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 771a919ccd0bd75fe6197299ae057647ece89a7e

  ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java 9f498c7fb88a7a9f77b8c6739c097a2b26b0c617

  ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
e6ec44504685bd9e53f158cc359b8a7b79fd0166 

Diff: https://reviews.apache.org/r/54393/diff/


Testing
-------

All itests/hive-blobstore tests run.

Added new blobstore tests:
- insert_into_dynamic_partitions.q
- insert_overwrite_dynamic_partitions.q

Waiting for HiveQA to run the rest of the q-tests.


Thanks,

Sergio Pena


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message