hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikram Dixit Kumaraswamy" <vikram.di...@gmail.com>
Subject Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging
Date Thu, 11 Sep 2014 19:00:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/#review53087
-----------------------------------------------------------

Ship it!


Ship It!

- Vikram Dixit Kumaraswamy


On Sept. 9, 2014, 7:32 a.m., Prasanth_J wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24627/
> -----------------------------------------------------------
> 
> (Updated Sept. 9, 2014, 7:32 a.m.)
> 
> 
> Review request for hive and Gunther Hagleitner.
> 
> 
> Bugs: HIVE-7704
>     https://issues.apache.org/jira/browse/HIVE-7704
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Currently tez falls back to MR task for merge file task. It will beneficial to convert
the merge file tasks to tez task to make use of the performance gains from tez.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9 
>   itests/src/test/resources/testconfiguration.properties 99049ca 
>   ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java
6f23575 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bbf3f6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java
6809c79 
>   ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION 
>   ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d 
>   ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 
>   ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION 
>   ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 11c7578 
>   ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 8de452f 
>   ql/src/test/results/clientpositive/list_bucket_dml_4.q.out b1c060e 
>   ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 3450d63 
>   ql/src/test/results/clientpositive/list_bucket_dml_7.q.out f6a4cb5 
>   ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 796c7af 
>   ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 
>   ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 
>   ql/src/test/results/clientpositive/orc_createas1.q.out 993c853 
>   ql/src/test/results/clientpositive/orc_merge1.q.out 7f88125 
>   ql/src/test/results/clientpositive/orc_merge3.q.out 258f538 
>   ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036 
>   ql/src/test/results/clientpositive/rcfile_merge1.q.out cadb2c6 
>   ql/src/test/results/clientpositive/rcfile_merge2.q.out 807b8fb 
>   ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5 
>   ql/src/test/results/clientpositive/tez/orc_merge1.q.out e88bb6d 
>   ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68 
>   ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498 
>   ql/src/test/results/clientpositive/union_remove_12.q.out 033475f 
>   ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa 
>   ql/src/test/results/clientpositive/union_remove_14.q.out 315c958 
>   ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0 
>   ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2 
> 
> Diff: https://reviews.apache.org/r/24627/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message