Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAMJOb8=nJ6_n-j9qFix1=biYhaZgLbUW1zhbc5UiET7efnh-3w@mail.gmail.com>
References: 
 <CAKWX9VU+vVstw_Vu-yj61+=rptD8prX_-dVNKSMQDVrCsMuWNA@mail.gmail.com>
	<CAAOnQ7uC00E-4hrs8oG1Tn1u4+9-ng8DxCyfsc_sgdDBGrvqKQ@mail.gmail.com>
	<CABPQxsuZEuiYSwCxZqoCZBeiP2zKvtd7XnMJnwx+da5ybO+v4g@mail.gmail.com>
	<CAKWX9VUZqwvuCxiNM4-9fZd-e=vNM5xFQwkSS1YdyP7VaLuCYA@mail.gmail.com>
	<CAKWX9VXS4fGVSTHcmDqvpy3Ze7eO0jv=p7xwh1etRGZtaHW9AQ@mail.gmail.com>
	<CABPQxsv+Ss_V9FH-kXDXRa80FfS7g0CPpmVG-y-NGEAYEE=7Mw@mail.gmail.com>
	<CABPQxstGyLS_tMEEP8dxSHHrBLVuYs1PNgGXbsQv3kkg6kCq=w@mail.gmail.com>
	<CAKWX9VWRdtU9N3XAN7cVd3JGvG2omoVx=rQ4_n3Pz6Uwzh3jTw@mail.gmail.com>
	<CAKWX9VXjKpsEmJEqymf6Hy+hcbvruVFMJf+JzOqU=SvJGNkS8Q@mail.gmail.com>
	<CAAOnQ7tVEXRJaZ9EZ-ppRWRXwudKhaV3_m7rp0Fdu+GcABPvyA@mail.gmail.com>
	<CAGOvqipBpgo=6Snj-kDPV5KABPTeK6SUmqebv7Z2e+d7LdgEug@mail.gmail.com>
	<CAMJOb8=nJ6_n-j9qFix1=biYhaZgLbUW1zhbc5UiET7efnh-3w@mail.gmail.com>
Date: Thu, 7 Aug 2014 19:55:28 -0700
Message-ID: 
 <CAMJOb8n_3TR-3-MWKUV5s7pPpG_b=W1C4hM5hB8FT52y+6aELw@mail.gmail.com>
Subject: Re: replacement for SPARK_JAVA_OPTS
From: Andrew Or <andrew@databricks.com>
To: Gary Malouf <malouf.gary@gmail.com>
Cc: Marcelo Vanzin <vanzin@cloudera.com>, Cody Koeninger <cody@koeninger.org>,
	"dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=089e0158c444772dc205001556b4

--089e0158c444772dc205001556b4
Content-Type: text/plain; charset=UTF-8

@Cody I took a quick glance at the Mesos code and it appears that we
currently do not even pass extra java options to executors except in coarse
grained mode, and even in this mode we do not pass them to executors
correctly. I have filed a related JIRA here:
https://issues.apache.org/jira/browse/SPARK-2921. This is a somewhat
serious limitation and we will try to fix this for 1.1.

-Andrew


2014-08-07 19:42 GMT-07:00 Andrew Or <andrew@databricks.com>:

> Thanks Marcelo, I have moved the changes to a new PR to describe the
> problems more clearly: https://github.com/apache/spark/pull/1845
>
> @Gary Yeah, the goal is to get this into 1.1 as a bug fix.
>
>
> 2014-08-07 17:30 GMT-07:00 Gary Malouf <malouf.gary@gmail.com>:
>
> Can this be cherry-picked for 1.1 if everything works out?  In my opinion,
>> it could be qualified as a bug fix.
>>
>>
>> On Thu, Aug 7, 2014 at 5:47 PM, Marcelo Vanzin <vanzin@cloudera.com>
>> wrote:
>>
>> > Andrew has been working on a fix:
>> > https://github.com/apache/spark/pull/1770
>> >
>> > On Thu, Aug 7, 2014 at 2:35 PM, Cody Koeninger <cody@koeninger.org>
>> wrote:
>> > > Just wanted to check in on this, see if I should file a bug report
>> > > regarding the mesos argument propagation.
>> > >
>> > >
>> > > On Thu, Jul 31, 2014 at 8:35 AM, Cody Koeninger <cody@koeninger.org>
>> > wrote:
>> > >
>> > >> 1. I've tried with and without escaping equals sign, it doesn't
>> affect
>> > the
>> > >> results.
>> > >>
>> > >> 2. Yeah, exporting SPARK_SUBMIT_OPTS from spark-env.sh works for
>> getting
>> > >> system properties set in the local shell (although not for
>> executors).
>> > >>
>> > >> 3. We're using the default fine-grained mesos mode, not setting
>> > >> spark.mesos.coarse, so it doesn't seem immediately related to that
>> > ticket.
>> > >> Should I file a bug report?
>> > >>
>> > >>
>> > >> On Thu, Jul 31, 2014 at 1:33 AM, Patrick Wendell <pwendell@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> The third issue may be related to this:
>> > >>> https://issues.apache.org/jira/browse/SPARK-2022
>> > >>>
>> > >>> We can take a look at this during the bug fix period for the 1.1
>> > >>> release next week. If we come up with a fix we can backport it into
>> > >>> the 1.0 branch also.
>> > >>>
>> > >>> On Wed, Jul 30, 2014 at 11:31 PM, Patrick Wendell <
>> pwendell@gmail.com>
>> > >>> wrote:
>> > >>> > Thanks for digging around here. I think there are a few distinct
>> > issues.
>> > >>> >
>> > >>> > 1. Properties containing the '=' character need to be escaped.
>> > >>> > I was able to load properties fine as long as I escape the '='
>> > >>> > character. But maybe we should document this:
>> > >>> >
>> > >>> > == spark-defaults.conf ==
>> > >>> > spark.foo a\=B
>> > >>> > == shell ==
>> > >>> > scala> sc.getConf.get("spark.foo")
>> > >>> > res2: String = a=B
>> > >>> >
>> > >>> > 2. spark.driver.extraJavaOptions, when set in the properties file,
>> > >>> > don't affect the driver when running in client mode (always the
>> case
>> > >>> > for mesos). We should probably document this. In this case you
>> need
>> > to
>> > >>> > either use --driver-java-options or set SPARK_SUBMIT_OPTS.
>> > >>> >
>> > >>> > 3. Arguments aren't propagated on Mesos (this might be because of
>> the
>> > >>> > other issues, or a separate bug).
>> > >>> >
>> > >>> > - Patrick
>> > >>> >
>> > >>> > On Wed, Jul 30, 2014 at 3:10 PM, Cody Koeninger <
>> cody@koeninger.org>
>> > >>> wrote:
>> > >>> >> In addition, spark.executor.extraJavaOptions does not seem to
>> behave
>> > >>> as I
>> > >>> >> would expect; java arguments don't seem to be propagated to
>> > executors.
>> > >>> >>
>> > >>> >>
>> > >>> >> $ cat conf/spark-defaults.conf
>> > >>> >>
>> > >>> >> spark.master
>> > >>> >>
>> > >>>
>> >
>> mesos://zk://etl-01.mxstg:2181,etl-02.mxstg:2181,etl-03.mxstg:2181/masters
>> > >>> >> spark.executor.extraJavaOptions -Dfoo.bar.baz=23
>> > >>> >> spark.driver.extraJavaOptions -Dfoo.bar.baz=23
>> > >>> >>
>> > >>> >>
>> > >>> >> $ ./bin/spark-shell
>> > >>> >>
>> > >>> >> scala> sc.getConf.get("spark.executor.extraJavaOptions")
>> > >>> >> res0: String = -Dfoo.bar.baz=23
>> > >>> >>
>> > >>> >> scala> sc.parallelize(1 to 100).map{ i => (
>> > >>> >>      |  java.net.InetAddress.getLocalHost.getHostName,
>> > >>> >>      |  System.getProperty("foo.bar.baz")
>> > >>> >>      | )}.collect
>> > >>> >>
>> > >>> >> res1: Array[(String, String)] = Array((dn-01.mxstg,null),
>> > >>> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-01.mxstg,null),
>> > >>> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-01.mxstg,null),
>> > >>> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-01.mxstg,null),
>> > >>> >> (dn-01.mxstg,null), (dn-01.mxstg,null), (dn-02.mxstg,null),
>> > >>> >> (dn-02.mxstg,null), ...
>> > >>> >>
>> > >>> >>
>> > >>> >>
>> > >>> >> Note that this is a mesos deployment, although I wouldn't expect
>> > that
>> > >>> to
>> > >>> >> affect the availability of spark.driver.extraJavaOptions in a
>> local
>> > >>> spark
>> > >>> >> shell.
>> > >>> >>
>> > >>> >>
>> > >>> >> On Wed, Jul 30, 2014 at 4:18 PM, Cody Koeninger <
>> cody@koeninger.org
>> > >
>> > >>> wrote:
>> > >>> >>
>> > >>> >>> Either whitespace or equals sign are valid properties file
>> formats.
>> > >>> >>> Here's an example:
>> > >>> >>>
>> > >>> >>> $ cat conf/spark-defaults.conf
>> > >>> >>> spark.driver.extraJavaOptions -Dfoo.bar.baz=23
>> > >>> >>>
>> > >>> >>> $ ./bin/spark-shell -v
>> > >>> >>> Using properties file: /opt/spark/conf/spark-defaults.conf
>> > >>> >>> Adding default property:
>> > >>> spark.driver.extraJavaOptions=-Dfoo.bar.baz=23
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> scala>  System.getProperty("foo.bar.baz")
>> > >>> >>> res0: String = null
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> If you add double quotes, the resulting string value will have
>> > double
>> > >>> >>> quotes.
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> $ cat conf/spark-defaults.conf
>> > >>> >>> spark.driver.extraJavaOptions "-Dfoo.bar.baz=23"
>> > >>> >>>
>> > >>> >>> $ ./bin/spark-shell -v
>> > >>> >>> Using properties file: /opt/spark/conf/spark-defaults.conf
>> > >>> >>> Adding default property:
>> > >>> spark.driver.extraJavaOptions="-Dfoo.bar.baz=23"
>> > >>> >>>
>> > >>> >>> scala>  System.getProperty("foo.bar.baz")
>> > >>> >>> res0: String = null
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> Neither one of those affects the issue; the underlying problem
>> in
>> > my
>> > >>> case
>> > >>> >>> seems to be that bin/spark-class uses the SPARK_SUBMIT_OPTS and
>> > >>> >>> SPARK_JAVA_OPTS environment variables, but nothing parses
>> > >>> >>> spark-defaults.conf before the java process is started.
>> > >>> >>>
>> > >>> >>> Here's an example of the process running when only
>> > >>> spark-defaults.conf is
>> > >>> >>> being used:
>> > >>> >>>
>> > >>> >>> $ ps -ef | grep spark
>> > >>> >>>
>> > >>> >>> 514       5182  2058  0 21:05 pts/2    00:00:00 bash
>> > >>> ./bin/spark-shell -v
>> > >>> >>>
>> > >>> >>> 514       5189  5182  4 21:05 pts/2    00:00:22
>> > >>> /usr/local/java/bin/java
>> > >>> >>> -cp
>> > >>> >>>
>> > >>>
>> >
>> ::/opt/spark/conf:/opt/spark/lib/spark-assembly-1.0.1-hadoop2.3.0-mr1-cdh5.0.2.jar:/etc/hadoop/conf-mx
>> > >>> >>> -XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m
>> > >>> >>> org.apache.spark.deploy.SparkSubmit spark-shell -v --class
>> > >>> >>> org.apache.spark.repl.Main
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> Here's an example of it when the command line
>> > --driver-java-options is
>> > >>> >>> used (and thus things work):
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> $ ps -ef | grep spark
>> > >>> >>> 514       5392  2058  0 21:15 pts/2    00:00:00 bash
>> > >>> ./bin/spark-shell -v
>> > >>> >>> --driver-java-options -Dfoo.bar.baz=23
>> > >>> >>>
>> > >>> >>> 514       5399  5392 80 21:15 pts/2    00:00:06
>> > >>> /usr/local/java/bin/java
>> > >>> >>> -cp
>> > >>> >>>
>> > >>>
>> >
>> ::/opt/spark/conf:/opt/spark/lib/spark-assembly-1.0.1-hadoop2.3.0-mr1-cdh5.0.2.jar:/etc/hadoop/conf-mx
>> > >>> >>> -XX:MaxPermSize=128m -Dfoo.bar.baz=23 -Djava.library.path=
>> -Xms512m
>> > >>> >>> -Xmx512m org.apache.spark.deploy.SparkSubmit spark-shell -v
>> > >>> >>> --driver-java-options -Dfoo.bar.baz=23 --class
>> > >>> org.apache.spark.repl.Main
>> > >>> >>>
>> > >>> >>>
>> > >>> >>>
>> > >>> >>>
>> > >>> >>> On Wed, Jul 30, 2014 at 3:43 PM, Patrick Wendell <
>> > pwendell@gmail.com>
>> > >>> >>> wrote:
>> > >>> >>>
>> > >>> >>>> Cody - in your example you are using the '=' character, but in
>> our
>> > >>> >>>> documentation and tests we use a whitespace to separate the key
>> > and
>> > >>> >>>> value in the defaults file.
>> > >>> >>>>
>> > >>> >>>> docs: http://spark.apache.org/docs/latest/configuration.html
>> > >>> >>>>
>> > >>> >>>> spark.driver.extraJavaOptions -Dfoo.bar.baz=23
>> > >>> >>>>
>> > >>> >>>> I'm not sure if the java properties file parser will try to
>> > interpret
>> > >>> >>>> the equals sign. If so you might need to do this.
>> > >>> >>>>
>> > >>> >>>> spark.driver.extraJavaOptions "-Dfoo.bar.baz=23"
>> > >>> >>>>
>> > >>> >>>> Do those work for you?
>> > >>> >>>>
>> > >>> >>>> On Wed, Jul 30, 2014 at 1:32 PM, Marcelo Vanzin <
>> > vanzin@cloudera.com
>> > >>> >
>> > >>> >>>> wrote:
>> > >>> >>>> > Hi Cody,
>> > >>> >>>> >
>> > >>> >>>> > Could you file a bug for this if there isn't one already?
>> > >>> >>>> >
>> > >>> >>>> > For system properties SparkSubmit should be able to read
>> those
>> > >>> >>>> > settings and do the right thing, but that obviously won't
>> work
>> > for
>> > >>> >>>> > other JVM options... the current code should work fine in
>> > cluster
>> > >>> mode
>> > >>> >>>> > though, since the driver is a different process. :-)
>> > >>> >>>> >
>> > >>> >>>> >
>> > >>> >>>> > On Wed, Jul 30, 2014 at 1:12 PM, Cody Koeninger <
>> > >>> cody@koeninger.org>
>> > >>> >>>> wrote:
>> > >>> >>>> >> We were previously using SPARK_JAVA_OPTS to set java system
>> > >>> properties
>> > >>> >>>> via
>> > >>> >>>> >> -D.
>> > >>> >>>> >>
>> > >>> >>>> >> This was used for properties that varied on a
>> > >>> >>>> per-deployment-environment
>> > >>> >>>> >> basis, but needed to be available in the spark shell and
>> > workers.
>> > >>> >>>> >>
>> > >>> >>>> >> On upgrading to 1.0, we saw that SPARK_JAVA_OPTS had been
>> > >>> deprecated,
>> > >>> >>>> and
>> > >>> >>>> >> replaced by spark-defaults.conf and command line arguments
>> to
>> > >>> >>>> spark-submit
>> > >>> >>>> >> or spark-shell.
>> > >>> >>>> >>
>> > >>> >>>> >> However, setting spark.driver.extraJavaOptions and
>> > >>> >>>> >> spark.executor.extraJavaOptions in spark-defaults.conf is
>> not a
>> > >>> >>>> replacement
>> > >>> >>>> >> for SPARK_JAVA_OPTS:
>> > >>> >>>> >>
>> > >>> >>>> >>
>> > >>> >>>> >> $ cat conf/spark-defaults.conf
>> > >>> >>>> >> spark.driver.extraJavaOptions=-Dfoo.bar.baz=23
>> > >>> >>>> >>
>> > >>> >>>> >> $ ./bin/spark-shell
>> > >>> >>>> >>
>> > >>> >>>> >> scala> System.getProperty("foo.bar.baz")
>> > >>> >>>> >> res0: String = null
>> > >>> >>>> >>
>> > >>> >>>> >>
>> > >>> >>>> >> $ ./bin/spark-shell --driver-java-options "-Dfoo.bar.baz=23"
>> > >>> >>>> >>
>> > >>> >>>> >> scala> System.getProperty("foo.bar.baz")
>> > >>> >>>> >> res0: String = 23
>> > >>> >>>> >>
>> > >>> >>>> >>
>> > >>> >>>> >> Looking through the shell scripts for spark-submit and
>> > >>> spark-class, I
>> > >>> >>>> can
>> > >>> >>>> >> see why this is; parsing spark-defaults.conf from bash
>> could be
>> > >>> >>>> brittle.
>> > >>> >>>> >>
>> > >>> >>>> >> But from an ergonomic point of view, it's a step back to go
>> > from a
>> > >>> >>>> >> set-it-and-forget-it configuration in spark-env.sh, to
>> > requiring
>> > >>> >>>> command
>> > >>> >>>> >> line arguments.
>> > >>> >>>> >>
>> > >>> >>>> >> I can solve this with an ad-hoc script to wrap spark-shell
>> with
>> > >>> the
>> > >>> >>>> >> appropriate arguments, but I wanted to bring the issue up to
>> > see
>> > >>> if
>> > >>> >>>> anyone
>> > >>> >>>> >> else had run into it,
>> > >>> >>>> >> or had any direction for a general solution (beyond parsing
>> > java
>> > >>> >>>> properties
>> > >>> >>>> >> files from bash).
>> > >>> >>>> >
>> > >>> >>>> >
>> > >>> >>>> >
>> > >>> >>>> > --
>> > >>> >>>> > Marcelo
>> > >>> >>>>
>> > >>> >>>
>> > >>> >>>
>> > >>>
>> > >>
>> > >>
>> >
>> >
>> >
>> > --
>> > Marcelo
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>> >
>> >
>>
>
>

--089e0158c444772dc205001556b4--