spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-20435) More thorough redaction of sensitive information from logs/UI, more unit tests
Date Sat, 22 Apr 2017 00:42:04 GMT


Apache Spark commented on SPARK-20435:

User 'markgrover' has created a pull request for this issue:

> More thorough redaction of sensitive information from logs/UI, more unit tests
> ------------------------------------------------------------------------------
>                 Key: SPARK-20435
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Mark Grover
> SPARK-18535 and SPARK-19720 were works to redact sensitive information (e.g. hadoop credential
provider password, AWS access/secret keys) from event logs + YARN logs + UI and from the console
output, respectively.
> While some unit tests were added along with these changes - they asserted when a sensitive
key was found, that redaction took place for that key. They didn't assert globally that when
running a full-fledged Spark app (whether or YARN or locally), that sensitive information
was not present in any of the logs or UI. Such a test would also prevent regressions from
happening in the future if someone unknowingly adds extra logging that publishes out sensitive
information to disk or UI.
> Consequently, it was found that in some Java configurations, sensitive information was
still being leaked in the event logs under the {{SparkListenerEnvironmentUpdate}} event, like
> {code}
> "":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password
> {code}
> "secret_password" should have been redacted.
> Moreover, previously redaction logic was only checking if the key matched the secret
regex pattern, it'd redact it's value. That worked for most cases. However, in the above case,
the key ( doesn't tell much, so the value needs to be searched. So the check
needs to be expanded to match against values as well.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message