From reviews-return-1004343-archive-asf-public=cust-asf.ponee.io@spark.apache.org Mon Jan 6 14:00:05 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 48F8618067C for ; Mon, 6 Jan 2020 15:00:05 +0100 (CET) Received: (qmail 9311 invoked by uid 500); 6 Jan 2020 14:00:04 -0000 Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@spark.apache.org Received: (qmail 9131 invoked by uid 99); 6 Jan 2020 14:00:04 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jan 2020 14:00:04 +0000 From: GitBox To: reviews@spark.apache.org Subject: [GitHub] [spark] durgaswaroop edited a comment on issue #5617: [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties in spark-shell and spark-submit Message-ID: <157831920395.8905.14562247108779957006.gitbox@gitbox.apache.org> References: In-Reply-To: Date: Mon, 06 Jan 2020 14:00:03 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit durgaswaroop edited a comment on issue #5617: [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties in spark-shell and spark-submit URL: https://github.com/apache/spark/pull/5617#issuecomment-571147738 I'm not entirely sure if this is related to this PR, but I am getting warnings with non-spark configuration. I'm using pyspark 2.4.3 with spark 2.4.3 and I'm trying to add configuration for a file system scheme. I'm adding the configuration as ``` SparkSession.builder \ .appName(..) \ .master(..) \ .config('fs.oci.client.hostname', '..') \ .config('fs.oci.client.auth.pemfilepath'. '..') \ .. ``` And with those, I get warnings as follows: ``` Warning: Ignoring non-spark config property: fs.oci.client.hostname=.... Warning: Ignoring non-spark config property: fs.oci.client.auth.fingerprint=.... Warning: Ignoring non-spark config property: fs.oci.client.auth.tenantId=.... Warning: Ignoring non-spark config property: fs.oci.client.auth.pemfilepath=.... ``` How to set non-spark config in pyspark? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org For additional commands, e-mail: reviews-help@spark.apache.org