spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu (Simon) Chen" <xche...@gmail.com>
Subject Re: spark 1.0.0 on yarn
Date Mon, 02 Jun 2014 01:37:49 GMT
That helped a bit... Now I have a different failure: the start up process
is stuck in an infinite loop outputting the following message:

14/06/02 01:34:56 INFO cluster.YarnClientSchedulerBackend: Application
report from ASM:
 appMasterRpcPort: -1
 appStartTime: 1401672868277
 yarnAppState: ACCEPTED

I am using the hadoop 2 prebuild package. Probably it doesn't have the
latest yarn client.

-Simon




On Sun, Jun 1, 2014 at 9:03 PM, Patrick Wendell <pwendell@gmail.com> wrote:

> As a debugging step, does it work if you use a single resource manager
> with the key "yarn.resourcemanager.address" instead of using two named
> resource managers? I wonder if somehow the YARN client can't detect
> this multi-master set-up.
>
> On Sun, Jun 1, 2014 at 12:49 PM, Xu (Simon) Chen <xchenum@gmail.com>
> wrote:
> > Note that everything works fine in spark 0.9, which is packaged in CDH5:
> I
> > can launch a spark-shell and interact with workers spawned on my yarn
> > cluster.
> >
> > So in my /opt/hadoop/conf/yarn-site.xml, I have:
> >     ...
> >     <property>
> >         <name>yarn.resourcemanager.address.rm1</name>
> >         <value>controller-1.mycomp.com:23140</value>
> >     </property>
> >     ...
> >     <property>
> >         <name>yarn.resourcemanager.address.rm2</name>
> >         <value>controller-2.mycomp.com:23140</value>
> >     </property>
> >     ...
> >
> > And the other usual stuff.
> >
> > So spark 1.0 is launched like this:
> > Spark Command: java -cp
> >
> ::/home/chenxu/spark-1.0.0-bin-hadoop2/conf:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/opt/hadoop/conf
> > -XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m
> > org.apache.spark.deploy.SparkSubmit spark-shell --master yarn-client
> --class
> > org.apache.spark.repl.Main
> >
> > I do see "/opt/hadoop/conf" included, but not sure it's the right place.
> >
> > Thanks..
> > -Simon
> >
> >
> >
> > On Sun, Jun 1, 2014 at 1:57 PM, Patrick Wendell <pwendell@gmail.com>
> wrote:
> >>
> >> I would agree with your guess, it looks like the yarn library isn't
> >> correctly finding your yarn-site.xml file. If you look in
> >> yarn-site.xml do you definitely the resource manager
> >> address/addresses?
> >>
> >> Also, you can try running this command with
> >> SPARK_PRINT_LAUNCH_COMMAND=1 to make sure the classpath is being
> >> set-up correctly.
> >>
> >> - Patrick
> >>
> >> On Sat, May 31, 2014 at 5:51 PM, Xu (Simon) Chen <xchenum@gmail.com>
> >> wrote:
> >> > Hi all,
> >> >
> >> > I tried a couple ways, but couldn't get it to work..
> >> >
> >> > The following seems to be what the online document
> >> > (http://spark.apache.org/docs/latest/running-on-yarn.html) is
> >> > suggesting:
> >> >
> >> >
> SPARK_JAR=hdfs://test/user/spark/share/lib/spark-assembly-1.0.0-hadoop2.2.0.jar
> >> > YARN_CONF_DIR=/opt/hadoop/conf ./spark-shell --master yarn-client
> >> >
> >> > Help info of spark-shell seems to be suggesting "--master yarn
> >> > --deploy-mode
> >> > cluster".
> >> >
> >> > But either way, I am seeing the following messages:
> >> > 14/06/01 00:33:20 INFO client.RMProxy: Connecting to ResourceManager
> at
> >> > /0.0.0.0:8032
> >> > 14/06/01 00:33:21 INFO ipc.Client: Retrying connect to server:
> >> > 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
> >> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> >> > 14/06/01 00:33:22 INFO ipc.Client: Retrying connect to server:
> >> > 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is
> >> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1
> SECONDS)
> >> >
> >> > My guess is that spark-shell is trying to talk to resource manager to
> >> > setup
> >> > spark master/worker nodes - I am not sure where 0.0.0.0:8032 came
> from
> >> > though. I am running CDH5 with two resource managers in HA mode. Their
> >> > IP/port should be in /opt/hadoop/conf/yarn-site.xml. I tried both
> >> > HADOOP_CONF_DIR and YARN_CONF_DIR, but that info isn't picked up.
> >> >
> >> > Any ideas? Thanks.
> >> > -Simon
> >
> >
>

Mime
View raw message