mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stratos Dimopoulos <stratos.dimopou...@gmail.com>
Subject Fwd: Mesos resource sharing between frameworks
Date Tue, 31 Mar 2015 01:02:15 GMT
Following up my earlier question...Do you have any suggestion on the best
way to configure Spark and Hadoop in order to equally share resources?

I see that for Hadoop I have the: mapred.mesos.total.map.slots.minimum
and mapred.mesos.total.reduce.slots.minimum but I cannot find a similar
property for the maximum slots.
Moreover I am not aware of similar configuration properties at Spark . (Is
this the most relevant? spark.scheduler.minRegisteredResourcesRatio)

Are you aware of a place that I can have documented a full list of
available configurations for both frameworks? So far my source is this:
https://github.com/mesos/hadoop/blob/master/configuration.md for hadoop and
this: http://spark.apache.org/docs/1.2.1/configuration.html#spark-properties
for spark but they don't include all the properties.

If you are not aware of any documentation maybe suggestion on the parts of
code that I could check to find these configuration parameters?

thanks,
Stratos

On Thu, Mar 19, 2015 at 7:32 PM, Stratos Dimopoulos <
stratos.dimopoulos@gmail.com> wrote:

> Thank you for the answer Benjamin!
>
> On Thu, Mar 19, 2015 at 6:05 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Guaranteeing fairness in such situations requires pre-emption of running
>> tasks/executors, which is not yet provided in mesos.
>>
>> For now, you can try reserving a minimum amount of resources for each
>> framework, note however that this may reduce your efficiency if you
>> over-estimate the minimum reservation needed. This is because reserved
>> resources cannot be used by other roles. In the future, to avoid the
>> efficiency issue, other roles will be able to use reserved resources in a
>> pre-emptable way.
>>
>> On Thu, Mar 19, 2015 at 3:39 PM, Stratos Dimopoulos <
>> stratos.dimopoulos@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Is there a way to dynamically fair-share resources between the
>>> frameworks that running on top of Mesos? What I would like to do is when
>>> running for example MapReduce and Spark together, allocate 50% of the
>>> resources to each of them (if they need them) and when running MapReduce,
>>> Spark, Storm 33% of the resources to each.
>>>
>>> What I observe is that when I am running Spark in fine grained mode and
>>> MapReduce together, MR will gradually take over all the resources as the
>>> task trackers of Spark are finished and in the meantime before Spark
>>> manages to stage new TaskTrackers Hadoop gets more and more resources until
>>> it actually takes over the whole cluster. The opposite happens with I run
>>> Spark in coarse grained mode. In this case Spark is faster staging the task
>>> trackers and it will manage to get 100% of the cluster before Hadoop gets
>>> any.
>>>
>>> I checked this:
>>> http://mesos.apache.org/documentation/latest/framework-rate-limiting/
>>> that might help but what I would really want to is to share the
>>> resources equally between registered frameworks. Any ideas?
>>>
>>> thanks!
>>>
>>> Stratos
>>>
>>
>>
>

Mime
View raw message