Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6F5561794D for ; Mon, 27 Oct 2014 23:03:55 +0000 (UTC) Received: (qmail 73639 invoked by uid 500); 27 Oct 2014 23:03:55 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 73568 invoked by uid 500); 27 Oct 2014 23:03:55 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 73552 invoked by uid 99); 27 Oct 2014 23:03:54 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Oct 2014 23:03:54 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 2C9A71DF5F9; Mon, 27 Oct 2014 23:03:59 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============3820262564201075575==" MIME-Version: 1.0 Subject: Re: Review Request 27247: convert joinOp to MapJoinOp and generate MapWorks only From: "Szehon Ho" To: "Szehon Ho" , "Xuefu Zhang" , "Rui Li" Cc: "hive" , "Suhas Satish" Date: Mon, 27 Oct 2014 23:03:59 -0000 Message-ID: <20141027230359.7143.52041@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Szehon Ho" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/27247/ X-Sender: "Szehon Ho" References: <20141027215304.7137.12364@reviews.apache.org> In-Reply-To: <20141027215304.7137.12364@reviews.apache.org> Reply-To: "Szehon Ho" X-ReviewRequest-Repository: hive-git --===============3820262564201075575== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27247/#review58712 ----------------------------------------------------------- Hi Suhas, thanks a lot for this great work! I haven't looked at all the details, but have some high-level comments just from the first pass. I think if we revert some of the SMB Join stuff and commented code that don't refer to mapjoin, it will be easier to see the core of this change. Thanks! ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java Would it make sense to rename it as SparkMapJoinOptimizer to go more with existing convention? I understand the convention is not very well doc'ed (or even followed). ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java Let's get rid of the commented code in this class (unless its necessary) to make it easier to read. ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java We can revert this class, as these changes are not consumed by mapjoin. ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java We can revert this class as methods are not being used for mapjoin. ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java We need to put the check of auto.convert.join before we enable this optimization. It would make it easier this way. ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java I don't see this class attached ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java Let's revert this code, as its not being used for mapjoin. - Szehon Ho On Oct. 27, 2014, 9:53 p.m., Suhas Satish wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27247/ > ----------------------------------------------------------- > > (Updated Oct. 27, 2014, 9:53 p.m.) > > > Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang. > > > Repository: hive-git > > > Description > ------- > > This addresses jira HIVE-8616 > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 46dcfaf > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkConvertJoinMapJoin.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java ed88c60 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 8e28887 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java 4f5feca > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 1c663c4 > > Diff: https://reviews.apache.org/r/27247/diff/ > > > Testing > ------- > > > Thanks, > > Suhas Satish > > --===============3820262564201075575==--