Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7830B11F0B for ; Wed, 13 Aug 2014 02:17:09 +0000 (UTC) Received: (qmail 31120 invoked by uid 500); 13 Aug 2014 02:17:09 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 31047 invoked by uid 500); 13 Aug 2014 02:17:09 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 31032 invoked by uid 99); 13 Aug 2014 02:17:08 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 02:17:08 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 74F701D9D02; Wed, 13 Aug 2014 02:17:08 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============0491492249276236265==" MIME-Version: 1.0 Subject: Re: Review Request 24627: HIVE-7704: Create tez task for fast file merging From: j.prasanth.j@gmail.com To: "Gunther Hagleitner" Cc: "hive" , j.prasanth.j@gmail.com Date: Wed, 13 Aug 2014 02:17:08 -0000 Message-ID: <20140813021708.1262.25345@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: noreply@reviews.apache.org X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/24627/ X-Sender: noreply@reviews.apache.org References: <20140813020942.1261.9873@reviews.apache.org> In-Reply-To: <20140813020942.1261.9873@reviews.apache.org> Reply-To: j.prasanth.j@gmail.com X-ReviewRequest-Repository: hive-git --===============0491492249276236265== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24627/ ----------------------------------------------------------- (Updated Aug. 13, 2014, 2:17 a.m.) Review request for hive and Gunther Hagleitner. Changes ------- Refreshed patch against trunk. Bugs: HIVE-7704 https://issues.apache.org/jira/browse/HIVE-7704 Repository: hive-git Description ------- Currently tez falls back to MR task for merge file task. It will beneficial to convert the merge file tasks to tez task to make use of the performance gains from tez. Diffs (updated) ----- itests/src/test/resources/testconfiguration.properties 62aa9a3 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cd017d8 ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java d5de58e ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java a2975cb ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 24dfed1 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1d6a93a ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 4e0fd79 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java e116426 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 8513e33 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java 31f3bcd ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileMapRecordProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/OrcMergeFileMapRecordProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/OrcMergeFileTezProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RCFileMergeFileMapRecordProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 1577827 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java c2ba782 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 951e918 ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/RCFileMergeFileTezProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java bf44548 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java 4651920 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java beb4f7d ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java a3ce699 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c437dd0 ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java b36152a ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java a6c92fb ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 76b4d03 ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java 6809c79 ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java dee6b1c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java 8513f99 ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java d58c59d ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION Diff: https://reviews.apache.org/r/24627/diff/ Testing ------- Thanks, Prasanth_J --===============0491492249276236265==--