JIMMY MCER= LAIN

DATA SCIENTIST (NERD)=

. . . . . .= . . . . . . . . . . . .

IF WE CAN= =E2=80=99T DOUBLE YOUR SALES,

ONE OF US= IS IN THE WRONG BUSINESS.

M: <= u>510.303.7751

On Tue, Oct 14, 2014 at 4:59 AM, Christophe Pr= =C3=A9aud <christophe.preaud@kelkoo.com> wrote:

Hello,

I have already posted a message with the exact same problem, and proposed a= patch (the subject is "Application failure in yarn-cluster mode"= ).
Can you test it, and see if it works for you?
I would be glad too if someone can confirm that it is a bug in Spark 1.1.0.=

Regards,
Christophe.

On 14/10/2014 03:15, Jimmy McErlain wrote:

BTW this has always worked for me before until we upgraded= the cluster to Spark 1.1.1...
J

=E1=90=A7

JIMMY MCER= LAIN

DATA SCIENTIST (NERD)=

. . . . . . . . . . . . . . . . . .

IF WE CAN= =E2=80=99T DOUBLE YOUR SALES,

ONE OF US= IS IN THE WRONG BUSINESS.

E:&nbs= p;jimmy@sellpoints.com &= nbsp;

M: <= u>510.303.7751

On Mon, Oct 13, 2014 at 5:39 PM, HARIPRIYA AYYAL= ASOMAYAJULA <aharipriya92@gmail.com> wrote= :

Helo,

Can you check if the jar file is available in the target->sca= la-2.10 folder?

When you use sbt package to make the jar file, that is where the jar f= ile would be located.

The following command works well for me:

spark-submit --class =E2=80=9CClassname" --master yarn-clust= er jarfile(withcomplete path)

Can you try checking with this initially and later add other optio= ns?

On Mon, Oct 13, 2014 at 7:36 PM, Jimmy <jimmy@sellpoints.com> wrote:

Having the exact same error with the exact same jar.... Do you work fo= r Altiscale? :)

J

Sent from my iPhone

On Oct 13, 2014, at 5:33 PM, Andy Srine <andy.srine@gmail.com> wrote:

Hi Guys,

Spark rookie here.= I am getting a file not found exception on the --jars. This is on the yarn= cluster mode and I am running the following command on our recently upgrad= ed Spark 1.1.1 environment.

./bin/spark-submit= --verbose --master yarn --deploy-mode cluster --class myEngine --driver-me= mory 1g --driver-library-path /hadoop/share/hadoop/mapreduce/lib/hadoop-lzo= -0.4.18-201406111750.jar --executor-memory 5g --executor-cores 5 --jars /home/andy/spark/lib/joda-convert-1.2.jar --q= ueue default --num-executors 4 /home/andy/spark/lib/my-spark-lib_1.0.jar

Th= is is the error I am hitting. Any tips would be much appreciated. The file = permissions looks fine on my local disk.

14/10/13 22:49:39 = INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED
14/10/13 22:49:39 = INFO impl.AMRMClientImpl: Waiting for application to be successfully unregi= stered.

Exception in threa= d "Driver" java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(N= ative Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Na= tiveMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invok= e(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:60= 6)

at org.apache.spark.deploy.yarn.ApplicationMaster= $$anon$2.run(ApplicationMaster.scala:162)

Caused by: org.apache.spark.SparkException: Job aborted due to stage = failure: Task 3 in stage 1.0 failed 4 times, most recent failure: Lost task= 3.3 in stage 1.0 (TID 12, 122-67.vb2.company.com): java.io.FileNotFoundException: ./j= oda-convert-1.2.jar (Permission denied)

&nbs= p; java.io.FileOutputStream.open(Native Method)

&nbs= p; java.io.FileOutputStream.<init>(FileOutputStream.java:221)<= /p>
&nbs= p; com.google.common.io.Files$FileByteSink.openStream(Files.java:223= )

&nbs= p; com.google.common.io.Files$FileByteSink.openStream(Files.java:211= )

&nbs= p;

Thanks,
Andy

--

Regards,
Haripriya Ayyalasomayajula

Kelkoo SAS
Soci=C3=A9t=C3=A9 par Actions Simplifi=C3=A9e
Au capital de =E2=82=AC 4.168.964,30
Si=C3=A8ge social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pi=C3=A8ces jointes sont confidentiels et =C3=A9tablis = =C3=A0 l'attention exclusive de leurs destinataires. Si vous n'=C3=AAtes pa= s le destinataire de ce message, merci de le d=C3=A9truire et d'en avertir = l'exp=C3=A9diteur.