hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From j.prasant...@gmail.com
Subject Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging
Date Sat, 06 Sep 2014 02:03:57 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/
-----------------------------------------------------------

(Updated Sept. 6, 2014, 2:03 a.m.)


Review request for hive and Gunther Hagleitner.


Changes
-------

Fixed test failures


Bugs: HIVE-7704
    https://issues.apache.org/jira/browse/HIVE-7704


Repository: hive-git


Description
-------

Currently tez falls back to MR task for merge file task. It will beneficial to convert the
merge file tasks to tez task to make use of the performance gains from tez.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 
  itests/src/test/resources/testconfiguration.properties 99049ca 
  ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575

  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e 
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79

  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION 
  ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d 
  ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 
  ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION 
  ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36 
  ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac 
  ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5 
  ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb 
  ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db 
  ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc 
  ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 
  ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 
  ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd 
  ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206 
  ql/src/test/results/clientpositive/orc_merge3.q.out 258f538 
  ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036 
  ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd 
  ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3 
  ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5 
  ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878 
  ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68 
  ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498 
  ql/src/test/results/clientpositive/union_remove_12.q.out 033475f 
  ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa 
  ql/src/test/results/clientpositive/union_remove_14.q.out 315c958 
  ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0 
  ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2 

Diff: https://reviews.apache.org/r/24627/diff/


Testing
-------


Thanks,

Prasanth_J


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message