Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
Date: Thu, 2 Jul 2015 16:56:05 +0000 (UTC)
From: "Marco Massenzio (JIRA)" <jira@apache.org>
To: issues@mesos.apache.org
Message-ID: <JIRA.12842253.1435825831000.86550.1435856165420@Atlassian.JIRA>
In-Reply-To: <JIRA.12842253.1435825831000@Atlassian.JIRA>
References: <JIRA.12842253.1435825831000@Atlassian.JIRA>
 <JIRA.12842253.1435825831819@arcas>
Subject: [jira] [Commented] (MESOS-2985) Wrong spark.executor.memory when
 using different EC2 master and worker machine types
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MESOS-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612185#comment-14612185 ] 

Marco Massenzio commented on MESOS-2985:
----------------------------------------

Hi Stefano,

it would appear I was wrong on that: the PR was for the mesos/spark-ec2 github repo (I was instead talking about the main Mesos ASF repo) which they may very well support (I have no idea, I'm afraid) - so I'll leave it to others, more knowledgeable, folks to comment.

My gratitude for your fixing stuff still stands, though!
yes, life ain't easy :)


> Wrong spark.executor.memory when using different EC2 master and worker machine types
> ------------------------------------------------------------------------------------
>
>                 Key: MESOS-2985
>                 URL: https://issues.apache.org/jira/browse/MESOS-2985
>             Project: Mesos
>          Issue Type: Bug
>          Components: ec2
>            Reporter: Stefano Parmesan
>
> _(this is a mirror of [SPARK-8726|https://issues.apache.org/jira/browse/SPARK-8726])_
> By default, {{spark.executor.memory}} is set to the [min(slave_ram_kb, master_ram_kb);|https://github.com/mesos/spark-ec2/blob/e642aa362338e01efed62948ec0f063d5fce3242/deploy_templates.py#L32] when using the same instance type for master and workers you will not notice, but when using different ones (which makes sense, as the master cannot be a spot instance, and using a big machine for the master would be a waste of resources) the default amount of memory given to each worker is capped to the amount of RAM available on the master (ex: if you create a cluster with an m1.small master (1.7GB RAM) and one m1.large worker (7.5GB RAM), spark.executor.memory will be set to 512MB).


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)