spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aarondav <...@git.apache.org>
Subject [GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM
Date Fri, 07 Mar 2014 18:33:38 GMT
GitHub user aarondav opened a pull request:

    https://github.com/apache/spark/pull/99

    SPARK-929: Fully deprecate usage of SPARK_MEM

    (Continued from old repo, prior discussion at https://github.com/apache/incubator-spark/pull/615)
    
    This patch cements our deprecation of the SPARK_MEM environment variable by replacing
it with three more specialized variables:
    SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_DRIVER_MEMORY
    
    The creation of the latter two variables means that we can safely set driver/job memory
without accidentally setting the executor memory. Neither is public.
    
    SPARK_EXECUTOR_MEMORY is only used by the Mesos scheduler (and set within SparkContext).
The proper way of configuring executor memory is through the "spark.executor.memory" property.
    
    SPARK_DRIVER_MEMORY is the new way of specifying the amount of memory run by jobs launched
by spark-class, without possibly affecting executor memory.
    
    Other memory considerations:
    - The repl's memory can be set through the "--drivermem" command-line option, which really
just sets SPARK_DRIVER_MEMORY.
    - run-example doesn't use spark-class, so the only way to modify examples' memory is actually
an unusual use of SPARK_JAVA_OPTS (which is normally overriden in all cases by spark-class).
    
    This patch also fixes a lurking bug where spark-shell misused spark-class (the first argument
is supposed to be the main class name, not java options), as well as a bug in the Windows
spark-class2.cmd. I have not yet tested this patch on either Windows or Mesos, however.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aarondav/spark sparkmem

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/99.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #99
    
----
commit 9df4c68262dac0edde1ec5bdd1fd065d2bf34e00
Author: Aaron Davidson <aaron@databricks.com>
Date:   2014-02-17T23:09:51Z

    SPARK-929: Fully deprecate usage of SPARK_MEM
    
    This patch cements our deprecation of the SPARK_MEM environment variable
    by replacing it with case-specific variables:
    SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_DRIVER_MEMORY
    
    The creation of the latter two variables means that we can safely
    set driver/job memory without accidentally setting the executor memory.
    Neither is public.
    
    SPARK_EXECUTOR_MEMORY is only used by the Mesos scheduler (and set
    within SparkContext). The proper way of configuring executor memory
    is through the "spark.executor.memory" property.
    
    SPARK_DRIVER_MEMORY is the new way of specifying the amount of memory
    run by jobs launched by spark-class, without possibly affecting executor
    memory.
    
    Other memory considerations:
    - The repl's memory can be set through the "--drivermem" command-line option,
      which really just sets SPARK_DRIVER_MEMORY.
    - run-example doesn't use spark-class, so the only way to modify examples'
      memory is actually an unusual use of SPARK_JAVA_OPTS (which is normally
      overriden in all cases by spark-class).
    
    This patch also fixes a lurking bug where spark-shell misused spark-class
    (the first argument is supposed to be the main class name, not java
    options).

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message