spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "guojh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-25880) user set some hadoop configurations can not work
Date Tue, 30 Oct 2018 08:54:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-25880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668263#comment-16668263
] 

guojh edited comment on SPARK-25880 at 10/30/18 8:53 AM:
---------------------------------------------------------

The root cause of this issue is that spark use the sparkContext's conf to construct the hadoop
conf which is use to broadcasted. So, We should filter the properties that startwith 'spark.hadoop.'


was (Author: gjhkael):
The root cause of this issue is that use the sparkContext's conf to construct the hadoop conf
which is use to broadcasted. So, We should filter the properties that startwith 'spark.hadoop.'

> user set some hadoop configurations can not work
> ------------------------------------------------
>
>                 Key: SPARK-25880
>                 URL: https://issues.apache.org/jira/browse/SPARK-25880
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, SQL
>    Affects Versions: 2.2.0
>            Reporter: guojh
>            Priority: Major
>
> When user set some hadoop configuration in spark-defaults.conf, for instance: 
> spark.hadoop.mapreduce.input.fileinputformat.split.maxsize   100000
> and then user use the spark-sql and use set command to overwrite this configuration,
but it can not cover the value which set in the file of spark-defaults.conf. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message