hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Pena <sergio.p...@cloudera.com>
Subject Re: Review Request 54393: HIVE-15361: INSERT dynamic partition on S3 fails with a MoveTask failure
Date Wed, 07 Dec 2016 17:05:30 GMT


> On Dec. 7, 2016, 4:43 p.m., Mohit Sabharwal wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 1765
> > <https://reviews.apache.org/r/54393/diff/2/?file=1578074#file1578074line1765>
> >
> >     can BlobStorageUtils.areOptimizationsEnabled(conf) return true for non blobstores
 ?

Yes. This only checks if the flag is enabled or not. Useful for users that do not want Bobstore
optimizations even if they use Blobstore paths.

Another check happens at the end of the method that verifies that the paths are Blobstore.


> On Dec. 7, 2016, 4:43 p.m., Mohit Sabharwal wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line 1842
> > <https://reviews.apache.org/r/54393/diff/2/?file=1578074#file1578074line1842>
> >
> >     throw RuntimeException

I don't think this is necessary. It is a non-check exception. The mergeMovePaths() method
should never be called with an invalid MoveWork (that's why shouldMergePaths() is called),
but I left it in case a user attempts to call it in a different way in a different method.


- Sergio


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54393/#review158345
-----------------------------------------------------------


On Dec. 7, 2016, 5:02 p.m., Sergio Pena wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54393/
> -----------------------------------------------------------
> 
> (Updated Dec. 7, 2016, 5:02 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15361
>     https://issues.apache.org/jira/browse/HIVE-15361
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Problem:
> - DynamicPartitionCtx and ListBucketingCtx objects weren't set on the new MoveWork created
when merging the two MoveWork objects from the ConditionalTask.
> 
> Solution
> - Set the DynamicPartitionCtx and ListBucketingCtx objects to the new MoveWork created
for the S3 optimization.
> 
> Other changes
> - Merge the MoveWork objects inside the createCondTask() method for better error handling.
> - Only merge the MoveWork related to the moveOnlyMoveTask. The MoveWork from the mergeAndMoveMoveTask
may cause other issues that are not correctly tested.
> - Two new private methods are added to check and merge the conditional input/output tasks
to the linked MoveWork.
> 
> 
> Diffs
> -----
> 
>   itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q
PRE-CREATION 
>   itests/hive-blobstore/src/test/queries/clientpositive/insert_into_table.q 25e2e7007ff539223d9244ca9822aa65d1441eb0

>   itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions.q
PRE-CREATION 
>   itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_table.q 846b2b113f09a74a3f05c13ffb56163e81dc1e8e

>   itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
PRE-CREATION 
>   itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 223cdf4d17e7fe9959a68f22a0aee6ea3872e2a9

>   itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
9b993a6568b63aa21bf3e644832c6347954f6f59 
>   itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
PRE-CREATION 
>   itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
81bcc7674727d68e881413dc083d4969013bd0fb 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java adc1188f09c8019a8aa60403d5813d6fa4509ceb

>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadDesc.java bcd3125ab4ad20c00fec565e5004ee200c0187d5

>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 9a868a04ce93d5c2ee75b5c6e96a1401cea93133

>   ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 771a919ccd0bd75fe6197299ae057647ece89a7e

>   ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java 9f498c7fb88a7a9f77b8c6739c097a2b26b0c617

>   ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
e6ec44504685bd9e53f158cc359b8a7b79fd0166 
> 
> Diff: https://reviews.apache.org/r/54393/diff/
> 
> 
> Testing
> -------
> 
> All itests/hive-blobstore tests run.
> 
> Added new blobstore tests:
> - insert_into_dynamic_partitions.q
> - insert_overwrite_dynamic_partitions.q
> 
> Waiting for HiveQA to run the rest of the q-tests.
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message