Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7934EFD4C for ; Mon, 2 Sep 2013 11:01:55 +0000 (UTC) Received: (qmail 10008 invoked by uid 500); 2 Sep 2013 11:01:54 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 9976 invoked by uid 500); 2 Sep 2013 11:01:53 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 9959 invoked by uid 500); 2 Sep 2013 11:01:52 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 9945 invoked by uid 99); 2 Sep 2013 11:01:52 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Sep 2013 11:01:52 +0000 Date: Mon, 2 Sep 2013 11:01:52 +0000 (UTC) From: "Hudson (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-4002) Fetch task aggregation for simple group by query MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756015#comment-13756015 ] Hudson commented on HIVE-4002: ------------------------------ FAILURE: Integrated in Hive-trunk-h0.21 #2303 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2303/]) HIVE-4002 Fetch task aggregation for simple group by query (Navis Ryu and Yin Huai via egc) (ecapriolo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1519306) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchAggregation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java * /hive/trunk/ql/src/test/queries/clientpositive/fetch_aggregation.q * /hive/trunk/ql/src/test/results/clientpositive/fetch_aggregation.q.out * /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml * /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml * /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml * /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java > Fetch task aggregation for simple group by query > ------------------------------------------------ > > Key: HIVE-4002 > URL: https://issues.apache.org/jira/browse/HIVE-4002 > Project: Hive > Issue Type: Improvement > Components: Query Processor > Reporter: Navis > Assignee: Navis > Priority: Minor > Fix For: 0.12.0 > > Attachments: HIVE-4002.D8739.1.patch, HIVE-4002.D8739.2.patch, HIVE-4002.D8739.3.patch, HIVE-4002.D8739.4.patch, HIVE-4002.patch > > > Aggregation queries with no group-by clause (for example, select count(*) from src) executes final aggregation in single reduce task. But it's too small even for single reducer because the most of UDAF generates just single row for map aggregation. If final fetch task can aggregate outputs from map tasks, shuffling time can be removed. > This optimization transforms operator tree something like, > TS-FIL-SEL-GBY1-RS-GBY2-SEL-FS + FETCH-TASK > into > TS-FIL-SEL-GBY1-FS + FETCH-TASK(GBY2-SEL-LS) > With the patch, time taken for auto_join_filters.q test reduced to 6 min (10 min, before). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira