Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1434711341 for ; Fri, 29 Aug 2014 06:44:27 +0000 (UTC) Received: (qmail 87269 invoked by uid 500); 29 Aug 2014 06:44:26 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 87189 invoked by uid 500); 29 Aug 2014 06:44:26 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 87166 invoked by uid 99); 29 Aug 2014 06:44:26 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Aug 2014 06:44:26 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id B020E1DBF16; Fri, 29 Aug 2014 06:44:25 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============3634349075213696726==" MIME-Version: 1.0 Subject: Re: Review Request 25176: HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch] From: "Na Yang" To: "Szehon Ho" , "Xuefu Zhang" , "Brock Noland" Cc: "Na Yang" , "hive" Date: Fri, 29 Aug 2014 06:44:25 -0000 Message-ID: <20140829064425.17468.65181@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Na Yang" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/25176/ X-Sender: "Na Yang" References: <20140828234242.17468.11961@reviews.apache.org> In-Reply-To: <20140828234242.17468.11961@reviews.apache.org> Reply-To: "Na Yang" X-ReviewRequest-Repository: hive-git --===============3634349075213696726== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/ ----------------------------------------------------------- (Updated Aug. 29, 2014, 6:44 a.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Changes ------- 1. add .q.out for TestCliDriver test for all new spark .q tests 2. update existing .q.out files because of plan change Bugs: HIVE-7870 https://issues.apache.org/jira/browse/HIVE-7870 Repository: hive-git Description ------- HIVE-7870: Insert overwrite table query does not generate correct task plan [Spark Branch] The cause of this problem is during spark/tez task generation, the union file sink operator are cloned to two new filesink operator. The linkedfilesinkdesc info for those new filesink operators are missing. In addition, the two new filesink operators also need to be linked together. Diffs (updated) ----- itests/src/test/resources/testconfiguration.properties 6393671 ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 5ddc16d ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 379a39c ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 76fc290 ql/src/test/queries/clientpositive/union_remove_spark_1.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_10.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_11.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_15.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_16.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_17.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_18.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_19.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_2.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_20.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_21.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_24.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_25.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_3.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_4.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_5.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_6.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_7.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_8.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_spark_9.q PRE-CREATION ql/src/test/results/clientpositive/spark/sample8.q.out c7e333b ql/src/test/results/clientpositive/spark/union10.q.out 20c681e ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 ql/src/test/results/clientpositive/spark/union29.q.out b9546ef ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 ql/src/test/results/clientpositive/spark/union33.q.out b89757f ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 ql/src/test/results/clientpositive/spark/union_remove_spark_1.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_10.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_11.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_15.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_16.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_17.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_18.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_19.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_2.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_20.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_21.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_24.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_25.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_3.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_4.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_5.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_6.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_7.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_8.q.out PRE-CREATION ql/src/test/results/clientpositive/spark/union_remove_spark_9.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_1.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_10.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_11.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_15.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_16.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_17.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_18.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_19.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_2.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_20.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_21.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_24.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_25.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_3.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_4.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_5.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_6.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_7.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_8.q.out PRE-CREATION ql/src/test/results/clientpositive/union_remove_spark_9.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25176/diff/ Testing ------- Thanks, Na Yang --===============3634349075213696726==--