spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Heller, Chris" <>
Subject Re: Removing the Mesos fine-grained mode
Date Thu, 19 Nov 2015 13:52:01 GMT
I was one that argued for fine-grain mode, and there is something I still appreciate about
how fine-grain mode operates in terms of the way one would define a Mesos framework. That
said, with dyn-allocation and Mesos support for both resource reservation, oversubscription
and revocation, I think the direction is clear that the coarse mode is the proper way forward,
and having the two code paths is just noise.


From: Iulian DragoČ™ <<>>
Date: Thursday, November 19, 2015 at 6:42 AM
To: "<>" <<>>
Subject: Removing the Mesos fine-grained mode

Hi all,

Mesos is the only cluster manager that has a fine-grained mode, but it's more often than not
problematic, and it's a maintenance burden. I'd like to suggest removing it in the 2.0 release.

A few reasons:

- code/maintenance complexity. The two modes duplicate a lot of functionality (and sometimes
code) that leads to subtle differences or bugs. See SPARK-10444<>
and also this thread<>
and MESOS-3202<>
- it's not widely used (Reynold's previous thread<>
got very few responses from people relying on it)
- similar functionality can be achieved with dynamic allocation + coarse-grained mode

I suggest that Spark 1.6 already issues a warning if it detects fine-grained use, with removal
in the 2.0 release.



View raw message