mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stratos Dimopoulos <>
Subject Fwd: Mesos resource sharing between frameworks
Date Tue, 31 Mar 2015 01:02:15 GMT
Following up my earlier question...Do you have any suggestion on the best
way to configure Spark and Hadoop in order to equally share resources?

I see that for Hadoop I have the:
and but I cannot find a similar
property for the maximum slots.
Moreover I am not aware of similar configuration properties at Spark . (Is
this the most relevant? spark.scheduler.minRegisteredResourcesRatio)

Are you aware of a place that I can have documented a full list of
available configurations for both frameworks? So far my source is this: for hadoop and
for spark but they don't include all the properties.

If you are not aware of any documentation maybe suggestion on the parts of
code that I could check to find these configuration parameters?


On Thu, Mar 19, 2015 at 7:32 PM, Stratos Dimopoulos <> wrote:

> Thank you for the answer Benjamin!
> On Thu, Mar 19, 2015 at 6:05 PM, Benjamin Mahler <
>> wrote:
>> Guaranteeing fairness in such situations requires pre-emption of running
>> tasks/executors, which is not yet provided in mesos.
>> For now, you can try reserving a minimum amount of resources for each
>> framework, note however that this may reduce your efficiency if you
>> over-estimate the minimum reservation needed. This is because reserved
>> resources cannot be used by other roles. In the future, to avoid the
>> efficiency issue, other roles will be able to use reserved resources in a
>> pre-emptable way.
>> On Thu, Mar 19, 2015 at 3:39 PM, Stratos Dimopoulos <
>>> wrote:
>>> Hi All,
>>> Is there a way to dynamically fair-share resources between the
>>> frameworks that running on top of Mesos? What I would like to do is when
>>> running for example MapReduce and Spark together, allocate 50% of the
>>> resources to each of them (if they need them) and when running MapReduce,
>>> Spark, Storm 33% of the resources to each.
>>> What I observe is that when I am running Spark in fine grained mode and
>>> MapReduce together, MR will gradually take over all the resources as the
>>> task trackers of Spark are finished and in the meantime before Spark
>>> manages to stage new TaskTrackers Hadoop gets more and more resources until
>>> it actually takes over the whole cluster. The opposite happens with I run
>>> Spark in coarse grained mode. In this case Spark is faster staging the task
>>> trackers and it will manage to get 100% of the cluster before Hadoop gets
>>> any.
>>> I checked this:
>>> that might help but what I would really want to is to share the
>>> resources equally between registered frameworks. Any ideas?
>>> thanks!
>>> Stratos

View raw message