Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AC73210C88 for ; Sat, 26 Oct 2013 20:20:33 +0000 (UTC) Received: (qmail 46298 invoked by uid 500); 26 Oct 2013 20:20:32 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 46027 invoked by uid 500); 26 Oct 2013 20:20:31 -0000 Mailing-List: contact dev-help@spark.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.incubator.apache.org Delivered-To: mailing list dev@spark.incubator.apache.org Received: (qmail 46019 invoked by uid 99); 26 Oct 2013 20:20:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Oct 2013 20:20:31 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of markus.losoi@gmail.com designates 209.85.214.44 as permitted sender) Received: from [209.85.214.44] (HELO mail-bk0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Oct 2013 20:20:26 +0000 Received: by mail-bk0-f44.google.com with SMTP id mz10so1275820bkb.3 for ; Sat, 26 Oct 2013 13:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:date:message-id:mime-version:content-type :content-transfer-encoding:thread-index:content-language; bh=e9DlZ7+V/B4geAhQ1Wb9K6BxolssMxNWcei3ddLhbH8=; b=MPoEVZGzuGX//NNqpsy9HGp8gNloACs0iGOpu/dYl4zxXMAV/6WpbQXASWUEJZbQlR 5+++f1hByHXgasy2pk5RF5DnjY38pkoCDzkKPuMCQq7xDD9Q9gV/Uuc/Mq1EYgUOlcZF GRsjiRvWe5e+URVafjoD8UMwGJMoJDHCmVIMD6kFd0G2KmrkB61aHQUmLifybAXIIQkY vZqQBNGmGst8RjdiZEvSCuL9ZIixL7YXQEMMwh5gMiFkN+JtuDzoaKOeIkVxeM905RLT 4/aldxcOiciVdT9ZqHsT/LqJcsojkhkcoth9VO8AuXJvEX7RjTtA3lZNO1wz3cmVwOna Xi0w== X-Received: by 10.205.37.10 with SMTP id tc10mr3047962bkb.9.1382818805338; Sat, 26 Oct 2013 13:20:05 -0700 (PDT) Received: from MLTPadE320 ([2001:708:30:1070:c4b8:9d20:e73c:c033]) by mx.google.com with ESMTPSA id kk2sm7317282bkb.10.2013.10.26.13.20.04 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 26 Oct 2013 13:20:05 -0700 (PDT) From: "Markus Losoi" To: Subject: A quick note about SPARK_WORKER_OPTS and ExecutorRunner Date: Sat, 26 Oct 2013 23:19:29 +0300 Message-ID: <526c23f5.82eccc0a.38af.7155@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac7SiKrevfhjpw0SQqWbkQ4jWbSh6w== Content-Language: fi X-Virus-Checked: Checked by ClamAV on apache.org Hi I was wondering why my SPARK_WORKER_OPTS at conf/spark-env.sh was not passed to the executors and noticed the following line in ExecutorRunner.scala (Spark 0.8.0): 116: val workerLocalOpts = Option(getenv("SPARK_JAVA_OPTS")).map(Utils.splitCommandString).getOrElse(Ni l) Is SPARK_JAVA_OPTS supposed to be SPARK_WORKER_OPTS in this line? The next line adds the options in SPARK_JAVA_OPTS: 117: val userOpts = getAppEnv("SPARK_JAVA_OPTS").map(Utils.splitCommandString).getOrElse(Nil) The options in both the variables workerLocalOpts and userOpts are aggregated into the executor options in the line: 126: Seq("-cp", classPath) ++ libraryOpts ++ workerLocalOpts ++ userOpts ++ memoryOpts Best regards, Markus Losoi (markus.losoi@gmail.com)