mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marco Massenzio (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-2985) Wrong spark.executor.memory when using different EC2 master and worker machine types
Date Thu, 02 Jul 2015 16:56:05 GMT

    [ https://issues.apache.org/jira/browse/MESOS-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612185#comment-14612185
] 

Marco Massenzio commented on MESOS-2985:
----------------------------------------

Hi Stefano,

it would appear I was wrong on that: the PR was for the mesos/spark-ec2 github repo (I was
instead talking about the main Mesos ASF repo) which they may very well support (I have no
idea, I'm afraid) - so I'll leave it to others, more knowledgeable, folks to comment.

My gratitude for your fixing stuff still stands, though!
yes, life ain't easy :)


> Wrong spark.executor.memory when using different EC2 master and worker machine types
> ------------------------------------------------------------------------------------
>
>                 Key: MESOS-2985
>                 URL: https://issues.apache.org/jira/browse/MESOS-2985
>             Project: Mesos
>          Issue Type: Bug
>          Components: ec2
>            Reporter: Stefano Parmesan
>
> _(this is a mirror of [SPARK-8726|https://issues.apache.org/jira/browse/SPARK-8726])_
> By default, {{spark.executor.memory}} is set to the [min(slave_ram_kb, master_ram_kb);|https://github.com/mesos/spark-ec2/blob/e642aa362338e01efed62948ec0f063d5fce3242/deploy_templates.py#L32]
when using the same instance type for master and workers you will not notice, but when using
different ones (which makes sense, as the master cannot be a spot instance, and using a big
machine for the master would be a waste of resources) the default amount of memory given to
each worker is capped to the amount of RAM available on the master (ex: if you create a cluster
with an m1.small master (1.7GB RAM) and one m1.large worker (7.5GB RAM), spark.executor.memory
will be set to 512MB).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message