spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davies Liu <dav...@databricks.com>
Subject Re: Spark-submit and multiple files
Date Fri, 20 Mar 2015 18:20:52 GMT
You MUST put --py-files BEFORE main.py, as mentioned in another threads.

On Fri, Mar 20, 2015 at 1:47 AM, Guillaume Charhon
<guillaume@databerries.com> wrote:
> Hi Davies,
>
> I am already using --py-files. The system does use the other file. The error
> I am getting is not trivial. Please check the error log.
>
>
>
> On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <davies@databricks.com> wrote:
>>
>> You could submit additional Python source via --py-files , for example:
>>
>> $ bin/spark-submit --py-files work.py main.py
>>
>> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez <guillaume@databerries.com>
>> wrote:
>> > Hello guys,
>> >
>> > I am having a hard time to understand how spark-submit behave with
>> > multiple
>> > files. I have created two code snippets. Each code snippet is composed
>> > of a
>> > main.py and work.py. The code works if I paste work.py then main.py in a
>> > pyspark shell. However both snippets do not work when using spark submit
>> > and
>> > generate different errors.
>> >
>> > Function add_1 definition outside
>> > http://www.codeshare.io/4ao8B
>> > https://justpaste.it/jzvj
>> >
>> > Embedded add_1 function definition
>> > http://www.codeshare.io/OQJxq
>> > https://justpaste.it/jzvn
>> >
>> > I am trying a way to make it work.
>> >
>> > Thank you for your support.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message