spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Heller, Chris" <chel...@akamai.com>
Subject Re: Removing the Mesos fine-grained mode
Date Thu, 19 Nov 2015 13:52:01 GMT
I was one that argued for fine-grain mode, and there is something I still appreciate about
how fine-grain mode operates in terms of the way one would define a Mesos framework. That
said, with dyn-allocation and Mesos support for both resource reservation, oversubscription
and revocation, I think the direction is clear that the coarse mode is the proper way forward,
and having the two code paths is just noise.

-Chris

From: Iulian DragoČ™ <iulian.dragos@typesafe.com<mailto:iulian.dragos@typesafe.com>>
Date: Thursday, November 19, 2015 at 6:42 AM
To: "dev@spark.apache.org<mailto:dev@spark.apache.org>" <dev@spark.apache.org<mailto:dev@spark.apache.org>>
Subject: Removing the Mesos fine-grained mode

Hi all,

Mesos is the only cluster manager that has a fine-grained mode, but it's more often than not
problematic, and it's a maintenance burden. I'd like to suggest removing it in the 2.0 release.

A few reasons:

- code/maintenance complexity. The two modes duplicate a lot of functionality (and sometimes
code) that leads to subtle differences or bugs. See SPARK-10444<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SPARK-2D10444&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=4_2dJBDiLqTcfXfX1LZluOo1U6tRKR2wKGGzfwiKdVY&e=>
and also this thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail-2Darchives.apache.org_mod-5Fmbox_spark-2Duser_201510.mbox_-253CCALxMP-2DA-2BaygNwSiyTM8ff20-2DMGWHykbhct94a2hwZTh1jWHp-5Fg-40mail.gmail.com-253E&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=SNFPzodGw7sgp3km9NKYM46gZHLguvxVNzCIeUlJzOw&e=>
and MESOS-3202<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_MESOS-2D3202&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=d-U4CohYsiZc0Zmj4KETn2dT_2ZFe5s3_IIbMm2tjJo&e=>
- it's not widely used (Reynold's previous thread<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Ddevelopers-2Dlist.1001551.n3.nabble.com_Please-2Dreply-2Dif-2Dyou-2Duse-2DMesos-2Dfine-2Dgrained-2Dmode-2Dtd14930.html&d=CwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=ylcFa5bBSUyTQqbx1Aqz47ec5BJJc7uk0YQ4EQKh-DY&m=36NeiiniCnBgPZ3AKAvvSJYBLQNxvpOcLoAi-VwXbtc&s=HGMiKyzxFDhpbomduKVIIRHWk9RDGDCk7tneJVQqTwo&e=>
got very few responses from people relying on it)
- similar functionality can be achieved with dynamic allocation + coarse-grained mode

I suggest that Spark 1.6 already issues a warning if it detects fine-grained use, with removal
in the 2.0 release.

Thoughts?

iulian

Mime
View raw message