spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (SPARK-1953) yarn client mode Application Master memory size is same as driver memory size
Date Fri, 09 Jan 2015 21:24:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or closed SPARK-1953.
----------------------------
          Resolution: Fixed
       Fix Version/s: 1.3.0
            Assignee: WangTaoTheTonic
    Target Version/s: 1.3.0  (was: 1.1.0)

> yarn client mode Application Master memory size is same as driver memory size
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-1953
>                 URL: https://issues.apache.org/jira/browse/SPARK-1953
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Thomas Graves
>            Assignee: WangTaoTheTonic
>             Fix For: 1.3.0
>
>
> With Spark on yarn in client mode, the application master that gets created to allocated
containers gets the same amount of memory as the driver running on the client. (--driver-memory
option through spark-submit)  This could definitely be more then what is really needed, thus
wasting resources.  The Application Master should be very small and require very little memory
since all its doing is allocating and starting containers.  
> We should allow the memory for the application master to be configurable separate from
the driver in client mode. 
>  We probably need to be careful about how we do this as to not cause confusion about
what the options do in the various modes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message