spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Diogo Munaro Vieira (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19720) Redact sensitive information from SparkSubmit console output
Date Wed, 26 Jul 2017 02:52:02 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101096#comment-16101096
] 

Diogo Munaro Vieira commented on SPARK-19720:
---------------------------------------------

Do you have plans to apply this fix in a Hadoop 2.5 compatible version of Spark?

Version 2.2 removed Hadoop 2.5 support

> Redact sensitive information from SparkSubmit console output
> ------------------------------------------------------------
>
>                 Key: SPARK-19720
>                 URL: https://issues.apache.org/jira/browse/SPARK-19720
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.2.0
>            Reporter: Mark Grover
>            Assignee: Mark Grover
>             Fix For: 2.2.0
>
>
> SPARK-18535 took care of redacting sensitive information from Spark event logs and UI.
However, it intentionally didn't bother redacting the same sensitive information from SparkSubmit's
console output because it was on the client's machine, which already had the sensitive information
on disk (in spark-defaults.conf) or on terminal (spark-submit command line).
> However, it seems now that it's better to redact information from SparkSubmit's console
output as well because orchestration software like Oozie usually expose SparkSubmit's console
output via a UI. To make matters worse, Oozie, in particular, always sets the {{--verbose}}
flag on SparkSubmit invocation, making the sensitive information readily available in its
UI (see [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248]
here).
> This is a JIRA for tracking redaction of sensitive information from SparkSubmit's console
output.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message