spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <>
Subject [jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode
Date Tue, 05 Sep 2017 14:34:00 GMT


Thomas Graves commented on SPARK-21888:

[~mgaido]  I don't think that is true unless something went into master that I'm not aware
of.  It doesn't work with 2.2 for sure.  We need the file/directory to get into the classpath
of the client submitting the application.  --files does work to get the driver and executors
to load it.  
if I'm missing something please let me know.

> Cannot add stuff to Client Classpath for Yarn Cluster Mode
> ----------------------------------------------------------
>                 Key: SPARK-21888
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Parth Gandhi
>            Priority: Minor
> While running Spark on Yarn in cluster mode, currently there is no way to add any config
files to Client classpath. An example for this is that suppose you want to run an application
that uses hbase. Then, unless and until we do not copy the necessary config files required
by hbase to Spark Config folder, we cannot specify or set their exact locations in classpath
on Client end which we could do so earlier by setting the environment variable "SPARK_CLASSPATH".

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message