spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: SparkSubmit and --driver-java-options
Date Wed, 30 Apr 2014 20:51:03 GMT
So I reproduced the problem here:

== test.sh ==
#!/bin/bash
for x in "$@"; do
  echo "arg: $x"
done
ARGS_COPY=$@
for x in "$ARGS_COPY"; do
  echo "arg_copy: $x"
done
==

./test.sh a b "c d e" f
arg: a
arg: b
arg: c d e
arg: f
arg_copy: a b c d e f

I'll dig around a bit more and see if we can fix it. Pretty sure we
aren't passing these argument arrays around correctly in bash.

On Wed, Apr 30, 2014 at 1:48 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:
> On Wed, Apr 30, 2014 at 1:41 PM, Patrick Wendell <pwendell@gmail.com> wrote:
>> Yeah I think the problem is that the spark-submit script doesn't pass
>> the argument array to spark-class in the right way, so any quoted
>> strings get flattened.
>>
>> I think we'll need to figure out how to do this correctly in the bash
>> script so that quoted strings get passed in the right way.
>
> I tried a few different approaches but finally ended up giving up; my
> bash-fu is apparently not strong enough. If you can make it work
> great, but I have "-J" working locally in case you give up like me.
> :-)
>
> --
> Marcelo

Mime
View raw message