hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From j.prasant...@gmail.com
Subject Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging
Date Wed, 13 Aug 2014 02:17:08 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24627/
-----------------------------------------------------------

(Updated Aug. 13, 2014, 2:17 a.m.)


Review request for hive and Gunther Hagleitner.


Changes
-------

Refreshed patch against trunk.


Bugs: HIVE-7704
    https://issues.apache.org/jira/browse/HIVE-7704


Repository: hive-git


Description
-------

Currently tez falls back to MR task for merge file task. It will beneficial to convert the
merge file tasks to tez task to make use of the performance gains from tez.


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties 62aa9a3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cd017d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java d5de58e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a2975cb 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 24dfed1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1d6a93a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 4e0fd79 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java e116426 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 8513e33 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java 31f3bcd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileMapRecordProcessor.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/OrcMergeFileMapRecordProcessor.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/OrcMergeFileTezProcessor.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RCFileMergeFileMapRecordProcessor.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 1577827 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java c2ba782 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 951e918 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/RCFileMergeFileTezProcessor.java PRE-CREATION

  ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java bf44548 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java beb4f7d 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c437dd0 
  ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java b36152a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 76b4d03 
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79

  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 8513f99 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java d58c59d 
  ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/24627/diff/


Testing
-------


Thanks,

Prasanth_J


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message