spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yinan Li (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-22743) Consolidate logic for handling spark.driver.memoryOverhead and spark.executor.memoryOverhead
Date Sat, 09 Dec 2017 01:34:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yinan Li updated SPARK-22743:
-----------------------------
    Description: In https://github.com/apache/spark/pull/19717, we will combine {{spark.yarn.[driver|executor].memoryOverhead}}
and {{spark.[driver|executor].driver.memoryOverhead}} into a single property {{spark.[driver|executor].memoryOverhead}}.
Both the Yarn mode and Kubernetes mode start to use the new properties, but the logic of handling
the property semantics is somehow duplicated in both modes. We should consolidate the logic
and make both modes share the logic.  (was: In https://github.com/apache/spark/pull/19717,
we will combine {{spark.yarn.{driver|executor}.memoryOverhead}} and {{spark.{driver|executor}.driver.memoryOverhead}}
into a single property {{spark.{driver|executor}.memoryOverhead}}. Both the Yarn mode and
Kubernetes mode start to use the new properties, but the logic of handling the property semantics
is somehow duplicated in both modes. We should consolidate the logic and make both modes share
the logic.)

> Consolidate logic for handling spark.driver.memoryOverhead and spark.executor.memoryOverhead
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22743
>                 URL: https://issues.apache.org/jira/browse/SPARK-22743
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 2.3.0
>            Reporter: Yinan Li
>
> In https://github.com/apache/spark/pull/19717, we will combine {{spark.yarn.[driver|executor].memoryOverhead}}
and {{spark.[driver|executor].driver.memoryOverhead}} into a single property {{spark.[driver|executor].memoryOverhead}}.
Both the Yarn mode and Kubernetes mode start to use the new properties, but the logic of handling
the property semantics is somehow duplicated in both modes. We should consolidate the logic
and make both modes share the logic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message