spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: SparkSubmit and --driver-java-options
Date Wed, 30 Apr 2014 20:51:03 GMT
So I reproduced the problem here:

== ==
for x in "$@"; do
  echo "arg: $x"
for x in "$ARGS_COPY"; do
  echo "arg_copy: $x"

./ a b "c d e" f
arg: a
arg: b
arg: c d e
arg: f
arg_copy: a b c d e f

I'll dig around a bit more and see if we can fix it. Pretty sure we
aren't passing these argument arrays around correctly in bash.

On Wed, Apr 30, 2014 at 1:48 PM, Marcelo Vanzin <> wrote:
> On Wed, Apr 30, 2014 at 1:41 PM, Patrick Wendell <> wrote:
>> Yeah I think the problem is that the spark-submit script doesn't pass
>> the argument array to spark-class in the right way, so any quoted
>> strings get flattened.
>> I think we'll need to figure out how to do this correctly in the bash
>> script so that quoted strings get passed in the right way.
> I tried a few different approaches but finally ended up giving up; my
> bash-fu is apparently not strong enough. If you can make it work
> great, but I have "-J" working locally in case you give up like me.
> :-)
> --
> Marcelo

View raw message