Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 14D9BD954 for ; Wed, 27 Feb 2013 11:21:17 +0000 (UTC) Received: (qmail 83101 invoked by uid 500); 27 Feb 2013 11:21:16 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 82834 invoked by uid 500); 27 Feb 2013 11:21:15 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 82538 invoked by uid 500); 27 Feb 2013 11:21:14 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 82528 invoked by uid 99); 27 Feb 2013 11:21:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2013 11:21:14 +0000 Date: Wed, 27 Feb 2013 11:21:14 +0000 (UTC) From: "Amareshwari Sriramadasu (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588233#comment-13588233 ] Amareshwari Sriramadasu commented on HIVE-3952: ----------------------------------------------- Tried out the patch, when we run query like the following : INSERT OVERWRITE DIRECTORY "/dir" Select .... It fails with exception : {noformat} java.lang.ClassCastException: org.apache.hadoop.hive.ql.exec.MoveTask cannot be cast to org.apache.hadoop.hive.ql.exec.MapRedTask at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver$CommonJoinTaskDispatcher.mayBeMergeMapJoinTaskWithMapReduceTask(CommonJoinResolver.java:291) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver$CommonJoinTaskDispatcher.processCurrentTask(CommonJoinResolver.java:535) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver$CommonJoinTaskDispatcher.dispatch(CommonJoinResolver.java:701) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:194) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:139) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:113) at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:79) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:8138) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8470) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:259) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:898) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) {noformat} > merge map-job followed by map-reduce job > ---------------------------------------- > > Key: HIVE-3952 > URL: https://issues.apache.org/jira/browse/HIVE-3952 > Project: Hive > Issue Type: Improvement > Components: Query Processor > Reporter: Namit Jain > Assignee: Vinod Kumar Vavilapalli > Attachments: HIVE-3952-20130226.txt > > > Consider the query like: > select count(*) FROM > ( select idOne, idTwo, value FROM > bigTable > JOIN > smallTableOne on (bigTable.idOne = smallTableOne.idOne) > ) firstjoin > JOIN > smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); > where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and > hive.auto.convert.join.noconditionaltask is set to true. > The joins are collapsed into mapjoins, and it leads to a map-only job > (for the map-joins) followed by a map-reduce job (for the group by). > Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira