spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] durgaswaroop edited a comment on issue #5617: [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties in spark-shell and spark-submit
Date Mon, 06 Jan 2020 14:00:03 GMT
durgaswaroop edited a comment on issue #5617: [SPARK-7037] [CORE] Inconsistent behavior for
non-spark config properties in spark-shell and spark-submit
URL: https://github.com/apache/spark/pull/5617#issuecomment-571147738
 
 
   I'm not entirely sure if this is related to this PR, but I am getting warnings with non-spark
configuration. 
   I'm using pyspark 2.4.3 with spark 2.4.3 and I'm trying to add configuration for a file
system scheme.
   
   I'm adding the configuration as 
   ```
   SparkSession.builder  \
                      .appName(..) \
                      .master(..) \
                      .config('fs.oci.client.hostname', '..')  \
                      .config('fs.oci.client.auth.pemfilepath'. '..') \
                      ..
   ``` 
   And with those, I get warnings as follows:
   ```
   Warning: Ignoring non-spark config property: fs.oci.client.hostname=....
   Warning: Ignoring non-spark config property: fs.oci.client.auth.fingerprint=....
   Warning: Ignoring non-spark config property: fs.oci.client.auth.tenantId=....
   Warning: Ignoring non-spark config property: fs.oci.client.auth.pemfilepath=....
   ```
   How to set non-spark config in pyspark?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message