Return-Path: X-Original-To: apmail-ignite-commits-archive@minotaur.apache.org Delivered-To: apmail-ignite-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B84BF18C95 for ; Fri, 14 Aug 2015 16:38:07 +0000 (UTC) Received: (qmail 90168 invoked by uid 500); 14 Aug 2015 16:38:07 -0000 Delivered-To: apmail-ignite-commits-archive@ignite.apache.org Received: (qmail 90138 invoked by uid 500); 14 Aug 2015 16:38:07 -0000 Mailing-List: contact commits-help@ignite.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.incubator.apache.org Delivered-To: mailing list commits@ignite.incubator.apache.org Received: (qmail 90129 invoked by uid 99); 14 Aug 2015 16:38:07 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2015 16:38:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E8D90DDDD1 for ; Fri, 14 Aug 2015 16:38:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.781 X-Spam-Level: X-Spam-Status: No, score=0.781 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 61rtO6LVNm3k for ; Fri, 14 Aug 2015 16:38:06 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id 48F1C25425 for ; Fri, 14 Aug 2015 16:37:57 +0000 (UTC) Received: (qmail 89642 invoked by uid 99); 14 Aug 2015 16:37:55 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2015 16:37:55 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 5FD07E6822; Fri, 14 Aug 2015 16:37:55 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: av@apache.org To: commits@ignite.incubator.apache.org Date: Fri, 14 Aug 2015 16:38:04 -0000 Message-Id: <3ee2277d98ba40488d4be4e7e964d560@git.apache.org> In-Reply-To: <4374a6f4d3d949b9a1c2097c41c5b667@git.apache.org> References: <4374a6f4d3d949b9a1c2097c41c5b667@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [10/34] incubator-ignite git commit: # ignite-1198 # ignite-1198 Project: http://git-wip-us.apache.org/repos/asf/incubator-ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-ignite/commit/2173b0e1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-ignite/tree/2173b0e1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-ignite/diff/2173b0e1 Branch: refs/heads/ignite-1093 Commit: 2173b0e1500d3dbc198a610042fca9b92d4f7de7 Parents: 023ffe0 Author: Alexey Goncharuk Authored: Wed Aug 5 14:05:07 2015 -0700 Committer: Alexey Goncharuk Committed: Wed Aug 5 14:25:13 2015 -0700 ---------------------------------------------------------------------- .../org/apache/ignite/spark/IgniteContext.scala | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-ignite/blob/2173b0e1/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala ---------------------------------------------------------------------- diff --git a/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala b/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala index 5267244a..6e48017 100644 --- a/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala +++ b/modules/spark/src/main/scala/org/apache/ignite/spark/IgniteContext.scala @@ -19,7 +19,8 @@ package org.apache.ignite.spark import org.apache.ignite.internal.IgnitionEx -import org.apache.ignite.{Ignition, Ignite} +import org.apache.ignite.internal.util.IgniteUtils +import org.apache.ignite.{IgniteSystemProperties, Ignition, Ignite} import org.apache.ignite.configuration.{CacheConfiguration, IgniteConfiguration} import org.apache.spark.{Logging, SparkContext} import org.apache.spark.sql.SQLContext @@ -41,8 +42,12 @@ class IgniteContext[K, V]( private val cfgClo = new Once(cfgF) + private val igniteHome = IgniteUtils.getIgniteHome + if (!client) { - val workers = sparkContext.getExecutorStorageStatus.length - 1 + // Get required number of executors with default equals to number of available executors. + val workers = sparkContext.getConf.getInt("spark.executor.instances", + sparkContext.getExecutorStorageStatus.length) if (workers <= 0) throw new IllegalStateException("No Spark executors found to start Ignite nodes.") @@ -125,6 +130,16 @@ class IgniteContext[K, V]( * @return Ignite instance. */ def ignite(): Ignite = { + val home = IgniteUtils.getIgniteHome + + if (home == null && igniteHome != null) { + logInfo("Setting IGNITE_HOME from driver not as it is not available on this worker: " + igniteHome) + + IgniteUtils.nullifyHomeDirectory() + + System.setProperty(IgniteSystemProperties.IGNITE_HOME, igniteHome) + } + val igniteCfg = cfgClo() try {