Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D10FF11D67 for ; Sat, 6 Sep 2014 02:03:58 +0000 (UTC) Received: (qmail 66012 invoked by uid 500); 6 Sep 2014 02:03:58 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 65931 invoked by uid 500); 6 Sep 2014 02:03:58 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 65918 invoked by uid 99); 6 Sep 2014 02:03:58 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Sep 2014 02:03:58 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 249371DD4E6; Sat, 6 Sep 2014 02:03:57 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============7887695754347898973==" MIME-Version: 1.0 Subject: Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging From: j.prasanth.j@gmail.com To: "Gunther Hagleitner" Cc: "hive" , j.prasanth.j@gmail.com Date: Sat, 06 Sep 2014 02:03:57 -0000 Message-ID: <20140906020357.13389.59967@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: noreply@reviews.apache.org X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/24627/ X-Sender: noreply@reviews.apache.org References: <20140905075151.16961.75333@reviews.apache.org> In-Reply-To: <20140905075151.16961.75333@reviews.apache.org> Reply-To: j.prasanth.j@gmail.com X-ReviewRequest-Repository: hive-git --===============7887695754347898973== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24627/ ----------------------------------------------------------- (Updated Sept. 6, 2014, 2:03 a.m.) Review request for hive and Gunther Hagleitner. Changes ------- Fixed test failures Bugs: HIVE-7704 https://issues.apache.org/jira/browse/HIVE-7704 Repository: hive-git Description ------- Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez. Diffs (updated) ----- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 itests/src/test/resources/testconfiguration.properties 99049ca ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575 ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79 ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36 ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5 ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206 ql/src/test/results/clientpositive/orc_merge3.q.out 258f538 ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036 ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3 ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5 ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878 ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68 ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498 ql/src/test/results/clientpositive/union_remove_12.q.out 033475f ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa ql/src/test/results/clientpositive/union_remove_14.q.out 315c958 ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0 ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2 Diff: https://reviews.apache.org/r/24627/diff/ Testing ------- Thanks, Prasanth_J --===============7887695754347898973==--