Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E83AF1018D for ; Thu, 16 Oct 2014 23:57:34 +0000 (UTC) Received: (qmail 43593 invoked by uid 500); 16 Oct 2014 23:57:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 43472 invoked by uid 500); 16 Oct 2014 23:57:34 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 43238 invoked by uid 500); 16 Oct 2014 23:57:34 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 43209 invoked by uid 99); 16 Oct 2014 23:57:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Oct 2014 23:57:34 +0000 Date: Thu, 16 Oct 2014 23:57:33 +0000 (UTC) From: "Chao (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-8436) Modify SparkWork to split works with multiple child works [Spark Branch] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174494#comment-14174494 ] Chao commented on HIVE-8436: ---------------------------- Yeah, let me add such a test with this patch. Originally I was thinking to do it as a followup, but now I think it's better to do it together, to ensure correctness. Also, my latest patch doesn't trigger the tests. Why is that? > Modify SparkWork to split works with multiple child works [Spark Branch] > ------------------------------------------------------------------------ > > Key: HIVE-8436 > URL: https://issues.apache.org/jira/browse/HIVE-8436 > Project: Hive > Issue Type: Sub-task > Components: Spark > Reporter: Xuefu Zhang > Assignee: Chao > Attachments: HIVE-8436.1-spark.patch, HIVE-8436.2-spark.patch, HIVE-8436.3-spark.patch > > > Based on the design doc, we need to split the operator tree of a work in SparkWork if the work is connected to multiple child works. The way splitting the operator tree is performed by cloning the original work and removing unwanted branches in the operator tree. Please refer to the design doc for details. > This process should be done right before we generate SparkPlan. We should have a utility method that takes the orignal SparkWork and return a modified SparkWork. > This process should also keep the information about the original work and its clones. Such information will be needed during SparkPlan generation (HIVE-8437). -- This message was sent by Atlassian JIRA (v6.3.4#6332)