spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shingo Furuyama (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-31432) bin/sbin scripts should allow to customize jars dir
Date Mon, 13 Apr 2020 07:23:00 GMT
Shingo Furuyama created SPARK-31432:
---------------------------------------

             Summary: bin/sbin scripts should allow to customize jars dir
                 Key: SPARK-31432
                 URL: https://issues.apache.org/jira/browse/SPARK-31432
             Project: Spark
          Issue Type: Improvement
          Components: Deploy
    Affects Versions: 2.4.5, 3.0.0
         Environment: In the script under bin/sbin, it is better that we can specify SPARK_JARS_DIR
as same as SPARK_CONF_DIR.

Our usecase:
We are trying to employ spark 2.4.5 with YARN in HDP2.6.4. Since there is an incompatible
conflict on commons-lang3 between spark 2.4.5 and HDP2.6.4, we tweak the jars by Maven Shade
Plugin.
The jars slightly differ from jars in spark 2.4.5, and we locate it in a directory different
from the default. So it is useful for us if we can set SPARK_JARS_DIR for bin/sbin scripts
to point the direcotry.
We can do that without the modification by deploying spark home as many as set of jars, but
it is somehow redundant.

Common usecase:
I believe there is a similer usecase. For example, deploying spark built for scala 2.11 and
2.12 in a machine and switch jars location by setting SPARK_JARS_DIR.
            Reporter: Shingo Furuyama






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message