spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-19720) Redact sensitive information from SparkSubmit console output
Date Thu, 02 Mar 2017 18:34:45 GMT

     [ https://issues.apache.org/jira/browse/SPARK-19720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin resolved SPARK-19720.
------------------------------------
       Resolution: Fixed
         Assignee: Mark Grover
    Fix Version/s: 2.2.0

> Redact sensitive information from SparkSubmit console output
> ------------------------------------------------------------
>
>                 Key: SPARK-19720
>                 URL: https://issues.apache.org/jira/browse/SPARK-19720
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.2.0
>            Reporter: Mark Grover
>            Assignee: Mark Grover
>             Fix For: 2.2.0
>
>
> SPARK-18535 took care of redacting sensitive information from Spark event logs and UI.
However, it intentionally didn't bother redacting the same sensitive information from SparkSubmit's
console output because it was on the client's machine, which already had the sensitive information
on disk (in spark-defaults.conf) or on terminal (spark-submit command line).
> However, it seems now that it's better to redact information from SparkSubmit's console
output as well because orchestration software like Oozie usually expose SparkSubmit's console
output via a UI. To make matters worse, Oozie, in particular, always sets the {{--verbose}}
flag on SparkSubmit invocation, making the sensitive information readily available in its
UI (see [code|https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L248]
here).
> This is a JIRA for tracking redaction of sensitive information from SparkSubmit's console
output.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message