Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: domain of
 spark.dubovsky.jakub@seznam.cz designates 77.75.72.26 as permitted sender)
From: <spark.dubovsky.jakub@seznam.cz>
To: <dev@spark.apache.org>
Subject: Protobuf version in mvn vs sbt
Date: Fri, 05 Dec 2014 21:51:32 +0100 (CET)
Message-Id: <Lhp.3c08L.spDiK0bKPE.1KWXhK@seznam.cz>
Mime-Version: 1.0 (szn-mime-2.0.1)
Content-Type: multipart/alternative;
	boundary="=_765c300a2ab380064db74ede=17919d76-71f0-58a2-a00e-19349e34696f_="

--=_765c300a2ab380064db74ede=17919d76-71f0-58a2-a00e-19349e34696f_=
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi devs,=0A=
=0A=
=C2=A0 I play with your amazing Spark here in Prague for some time. I have=
 =0A=
stumbled on a thing which I like to ask about. I create assembly jars from=
 =0A=
source and then use it to run simple jobs on our 2.3.0-cdh5.1.3 cluster =
=0A=
using yarn. Example of my usage [1]. Formerly I had started to use sbt for=
 =0A=
creating assemblies like this [2] which runs just fine. Then reading those=
 =0A=
maven-prefered stories here on dev list I found make-distribution.sh scrip=
t =0A=
in root of codebase and wanted to give it a try. I used it to create =
=0A=
assembly by both [3] and [4].=0A=
=0A=
=C2=A0 But I am not able to use assemblies created by make-distribution be=
cause =0A=
it refuses to be submited to cluster. Here is what happens:=0A=
- run [3] or [4]=0A=
- recompile app agains new assembly=0A=
- submit job using new assembly by [1] like command=0A=
- submit fails with important parts of stack trace being [5]=0A=
=0A=
=C2=A0 My guess is that it is due to improper version of protobuf included=
 in =0A=
assembly jar. My questions are:=0A=
- Can you confirm this hypothesis?=0A=
- What is the difference between sbt and mvn way of creating assembly? I =
=0A=
mean sbt works and mvn not...=0A=
- What additional option I need to pass to make-distribution to make it =
=0A=
work?=0A=
=0A=
=C2=A0 Any help/explanation here would be appreciated=0A=
=0A=
=C2=A0 Jakub=0A=
----------------------=0A=
[1] ./bin/spark-submit --num-executors 200 --master yarn-cluster --conf =
=0A=
spark.yarn.jar=3Dassembly/target/scala-2.10/spark-assembly-1.2.1-SNAPSHOT-=
=0A=
hadoop2.3.0-cdh5.1.3.jar --class org.apache.spark.mllib.=0A=
CreateGuidDomainDictionary root-0.1.jar ${args}=0A=
=0A=
[2] ./sbt/sbt -Dhadoop.version=3D2.3.0-cdh5.1.3 -Pyarn -Phive assembly/=
=0A=
assembly=0A=
=0A=
[3] ./make-distribution.sh -Dhadoop.version=3D2.3.0-cdh5.1.3 -Pyarn -Phive=
 -=0A=
DskipTests=0A=
=0A=
[4] ./make-distribution.sh -Dyarn.version=3D2.3.0 -Dhadoop.version=3D2.3.0=
-cdh=0A=
5.1.3 -Pyarn -Phive -DskipTests=0A=
=0A=
[5]Exception in thread "main" org.apache.hadoop.yarn.exceptions.=0A=
YarnRuntimeException: java.lang.reflect.InvocationTargetException=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.facto=
ries.impl.pb.RpcClientFactoryPBImpl.=0A=
getClient(RpcClientFactoryPBImpl.java:79)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.ipc.H=
adoopYarnProtoRPC.getProxy=0A=
(HadoopYarnProtoRPC.java:48)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.clien=
t.RMProxy$1.run(RMProxy.java:134)=0A=
...=0A=
Caused by: java.lang.reflect.InvocationTargetException=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeConstructo=
rAccessorImpl.newInstance0(Native =0A=
Method)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeConstructo=
rAccessorImpl.newInstance=0A=
(NativeConstructorAccessorImpl.java:39)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.DelegatingConstr=
uctorAccessorImpl.newInstance=0A=
(DelegatingConstructorAccessorImpl.java:27)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Constructo=
r.newInstance(Constructor.java:513)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.facto=
ries.impl.pb.RpcClientFactoryPBImpl.=0A=
getClient(RpcClientFactoryPBImpl.java:76)=0A=
... 27 more=0A=
Caused by: java.lang.VerifyError: class org.apache.hadoop.yarn.proto.=
=0A=
YarnServiceProtos$SubmitApplicationRequestProto overrides final method =
=0A=
getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoader.define=
Class1(Native Method)=0A=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoader.define=
ClassCond(ClassLoader.java:631)=0A=
=0A=
--=_765c300a2ab380064db74ede=17919d76-71f0-58a2-a00e-19349e34696f_=--