Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D528B104AC for ; Tue, 18 Nov 2014 06:48:34 +0000 (UTC) Received: (qmail 58135 invoked by uid 500); 18 Nov 2014 06:48:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 58071 invoked by uid 500); 18 Nov 2014 06:48:34 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 58059 invoked by uid 500); 18 Nov 2014 06:48:34 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 58056 invoked by uid 99); 18 Nov 2014 06:48:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Nov 2014 06:48:34 +0000 Date: Tue, 18 Nov 2014 06:48:34 +0000 (UTC) From: "Chengxiang Li (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-8835) identify dependency scope for Remote Spark Context.[Spark Branch] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated HIVE-8835: -------------------------------- Attachment: HIVE-8835.1-spark.patch Remote Spark Context need at least 3 dependencies to go through a query. # spark assembly jar # hive exec bundle jar # spark-client jar Hive ql module depends on spark-client module, we should add spark-client into hive exec bundle jar. > identify dependency scope for Remote Spark Context.[Spark Branch] > ----------------------------------------------------------------- > > Key: HIVE-8835 > URL: https://issues.apache.org/jira/browse/HIVE-8835 > Project: Hive > Issue Type: Sub-task > Components: Spark > Reporter: Chengxiang Li > Assignee: Chengxiang Li > Labels: Spark-M3 > Attachments: HIVE-8835.1-spark.patch > > > While submit job through Remote Spark Context, spark RDD graph generation and job submit is executed in remote side, so we have to add hive related dependency into its classpath with spark.driver.extraClassPath. instead of add all hive/hadoop dependency, we should narrow the scope and identify what dependency remote spark context required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)