Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2EBE8115C7 for ; Thu, 11 Sep 2014 19:00:31 +0000 (UTC) Received: (qmail 51919 invoked by uid 500); 11 Sep 2014 19:00:30 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 51841 invoked by uid 500); 11 Sep 2014 19:00:30 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 51790 invoked by uid 99); 11 Sep 2014 19:00:30 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Sep 2014 19:00:30 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id B4E041DD66D; Thu, 11 Sep 2014 19:00:26 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============6756809762477337233==" MIME-Version: 1.0 Subject: Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging From: "Vikram Dixit Kumaraswamy" To: "Gunther Hagleitner" Cc: "Vikram Dixit Kumaraswamy" , "hive" , j.prasanth.j@gmail.com Date: Thu, 11 Sep 2014 19:00:26 -0000 Message-ID: <20140911190026.23923.89418@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Vikram Dixit Kumaraswamy" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/24627/ X-Sender: "Vikram Dixit Kumaraswamy" References: <20140909073220.23924.11020@reviews.apache.org> In-Reply-To: <20140909073220.23924.11020@reviews.apache.org> Reply-To: "Vikram Dixit Kumaraswamy" X-ReviewRequest-Repository: hive-git --===============6756809762477337233== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24627/#review53087 ----------------------------------------------------------- Ship it! Ship It! - Vikram Dixit Kumaraswamy On Sept. 9, 2014, 7:32 a.m., Prasanth_J wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24627/ > ----------------------------------------------------------- > > (Updated Sept. 9, 2014, 7:32 a.m.) > > > Review request for hive and Gunther Hagleitner. > > > Bugs: HIVE-7704 > https://issues.apache.org/jira/browse/HIVE-7704 > > > Repository: hive-git > > > Description > ------- > > Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31aeba9 > itests/src/test/resources/testconfiguration.properties 99049ca > ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java 6f23575 > ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 > ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 > ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 > ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bbf3f6 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java 13ec642 > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb > ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e > ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e > ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79 > ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 > ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java PRE-CREATION > ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d > ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 > ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION > ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION > ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION > ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 11c7578 > ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 8de452f > ql/src/test/results/clientpositive/list_bucket_dml_4.q.out b1c060e > ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 3450d63 > ql/src/test/results/clientpositive/list_bucket_dml_7.q.out f6a4cb5 > ql/src/test/results/clientpositive/list_bucket_dml_9.q.out 796c7af > ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 > ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 > ql/src/test/results/clientpositive/orc_createas1.q.out 993c853 > ql/src/test/results/clientpositive/orc_merge1.q.out 7f88125 > ql/src/test/results/clientpositive/orc_merge3.q.out 258f538 > ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION > ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION > ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION > ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036 > ql/src/test/results/clientpositive/rcfile_merge1.q.out cadb2c6 > ql/src/test/results/clientpositive/rcfile_merge2.q.out 807b8fb > ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5 > ql/src/test/results/clientpositive/tez/orc_merge1.q.out e88bb6d > ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68 > ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498 > ql/src/test/results/clientpositive/union_remove_12.q.out 033475f > ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa > ql/src/test/results/clientpositive/union_remove_14.q.out 315c958 > ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0 > ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2 > > Diff: https://reviews.apache.org/r/24627/diff/ > > > Testing > ------- > > > Thanks, > > Prasanth_J > > --===============6756809762477337233==--