spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: multiple concurrent jobs
Date Tue, 19 Nov 2013 15:59:09 GMT
Ah, sorry -- misunderstood the question.


> On Nov 19, 2013, at 7:48 AM, Prashant Sharma <scrapcodes@gmail.com> wrote:
> 
> I think that is Scheduling Within an Application, and he asked across apps. Actually
spark standalone supports two ways of scheduling both are FIFO type. http://spark.incubator.apache.org/docs/latest/spark-standalone.html

> 
> One is spread out mode and the other is use as fewer node as possible [1] 
> 
> 1. https://github.com/apache/incubator-spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L383
> 
> 
> 
> 
> On Tue, Nov 19, 2013 at 9:02 PM, Mark Hamstra <mark@clearstorydata.com> wrote:
> >>
> >> According to the documentation, spark standalone currently only supports a FIFO
scheduling system.
> >
> >
> > That's not true.
> >
> > [sorry for the prior misfire]
> >  
> >
> >
> > On Tue, Nov 19, 2013 at 7:30 AM, Mark Hamstra <mark@clearstorydata.com> wrote:
> >>
> >>
> >>
> >>
> >> On Tue, Nov 19, 2013 at 6:50 AM, Yadid Ayzenberg <yadid@media.mit.edu>
wrote:
> >>>
> >>> Hi all,
> >>>
> >>> According to the documentation, spark standalone currently only supports
a FIFO scheduling system.
> >>> I understand its possible to limit the number of cores a job uses by setting
spark.cores.max.
> >>> When running a job, will spark try using the max number of cores on each
machine until it reaches the set limit, or will it do this round robin style - utilize a single
core on each machine -  if its already used a core on all of the slaves and the limit has
not been reached, spark will utilize an additional core on each machine and so on.
> >>>
> >>> I think the latter make more sense, but I want to be sure that is the case.
> >>>
> >>> Thanks,
> >>> Yadid
> >>>
> >>
> >
> 
> 
> 
> --
> s

Mime
View raw message