mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Dong <dongda...@gmail.com>
Subject Re: spark on mesos.
Date Fri, 27 Feb 2015 21:39:23 GMT
Hi, Dick,
  By Spark daemons I mean Master and Worker process running on master and
slaves respectively when you run sbin/start-master.sh and
./sbin/start-slaves.sh.

Cheers,
Dan


2015-02-27 15:02 GMT-06:00 Dick Davies <dick@hellooperator.net>:

> What do you mean by spark daemons?
>
> the spark shell (or any other spark application) acts as a Mesos framework,
> so until that's running spark isn't 'on' Mesos.
>
> On 27 February 2015 at 16:23, Dan Dong <dongdan39@gmail.com> wrote:
> > Hi, All,
> >   When I configured and started spark daemons, why I could not see it in
> > "Active Frameworks" on Mesos UI(for Hadoop, it will show up immediately
> when
> > hadoop daemons started)?. I can see the spark framework on mesos UI only
> > when I run the spark-shell command interactively. Is it normal?
> >
> > export
> > MESOS_NATIVE_LIBRARY=/home/ubuntu/mesos-0.21.0/build/lib/
> libmesos-0.21.0.so
> > export
> > PROTOBUF_JAR=/home/ubuntu/hadoop-2.5.0-cdh5.2.0/protobuf-java-2.5.0.jar
> > export MESOS_JAR=/home/ubuntu/hadoop-2.5.0-cdh5.2.0//mesos-0.21.0.jar
> > export
> >
> SPARK_EXECUTOR_URI=hdfs://clus-1:9000/user/ubuntu/spark-1.1.1-bin-hadoop2.4_mesos.tar.gz
> > export MASTER=mesos://clus-1:5050/mesos
> >
> >
> > Cheers,
> > Dan
> >
>

Mime
View raw message