mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haosdent <haosd...@gmail.com>
Subject Re: Unable to start more than two executors on Mesos with Spark
Date Sat, 13 Feb 2016 09:25:08 GMT
Hi, I think in coarse grain mode, Spark would schedule and allocate
resource by itself, just like what it do in YARN. So you only could see two
executors.
I think you could get more details from
http://spark.apache.org/docs/latest/running-on-mesos.html#mesos-run-modes


On Sat, Feb 13, 2016 at 4:16 AM, Rodrick Brown <rodrick@orchard-app.com>
wrote:

> I'm having an issue with Spark when running on Mesos in coarse grain mode
> no matter what options I use or how many free resources I have I only ever
> see two executors start.
>
> If I run Spark in fine grain mode on Mesos I see tasks eventually ramp up
> to the desired number I specified.
> Anyone know what could be causing this?
>
> I'm starting my job using the following commands --
>
> /usr/bin/timeout 3600 /opt/spark-1.5.1/bin/spark-submit --master "
> mesos://zk://prod-zookeeper-1.aws.xxx.com:2181,prod-zookeeper-2.aws.xxx.com:2181,prod-zookeeper-3.aws.xxx.com:2181/mesos"
> --conf spark.ui.port=31278 --conf spark.mesos.coarse=true --conf
> spark.mesos.extra.cores=3 --conf spark.cores.max=5 --conf
> spark.mesos.constraints="rack:spark" --conf spark.sql.tungsten.enabled=true
> --class
> com.orchard.dataloader.library.originators.prosper.LoadTrade_Prosper
> --total-executor-cores 3 --driver-memory 1536M --executor-memory 3011M
> --jars /data/orchard/etc/load-tradedata-accumulo-prod
> /data/orchard/jars/dataloader-library-59e0441cd06544442d2a84f5842b701e81097988-assembled.jar
>
>
> I'm using spark-1.5.1 and Mesos 0.26 Chronos-2.4.0 is starting the spark
> driver.
>
>
> If anyone have any ideas on what/where I should be focusing on to debug
> this?
>
> --
>
> *Rodrick Brown* / Systems Engineer
>
> +1 917 445 6839 / rodrick@orchardplatform.com
> <charlie@orchardplatform.com>
>
> *Orchard Platform*
>
> 101 5th Avenue, 4th Floor, New York, NY 10003
>
> http://www.orchardplatform.com
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending
> Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient of
> this communication, please delete it immediately and notify the sender by
> return email. Unauthorized reading, dissemination, distribution or copying
> of this communication is prohibited. This communication does not constitute
> an offer to sell or a solicitation of an indication of interest to purchase
> any loan, security or any other financial product or instrument, nor is it
> an offer to sell or a solicitation of an indication of interest to purchase
> any products or services to any persons who are prohibited from receiving
> such information under applicable law. The contents of this communication
> may not be accurate or complete and are subject to change without notice.
> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or completeness
> of the information contained herein. The intended recipient is advised to
> consult its own professional advisors, including those specializing in
> legal, tax and accounting matters. Orchard does not provide legal, tax or
> accounting advice.
>



-- 
Best Regards,
Haosdent Huang

Mime
View raw message