spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Grover (JIRA)" <>
Subject [jira] [Created] (SPARK-20435) More thorough redaction of sensitive information from logs/UI, more unit tests
Date Sat, 22 Apr 2017 00:35:04 GMT
Mark Grover created SPARK-20435:

             Summary: More thorough redaction of sensitive information from logs/UI, more
unit tests
                 Key: SPARK-20435
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.2.0
            Reporter: Mark Grover

SPARK-18535 and SPARK-19720 were works to redact sensitive information (e.g. hadoop credential
provider password, AWS access/secret keys) from event logs + YARN logs + UI and from the console
output, respectively.

While some unit tests were added along with these changes - they asserted when a sensitive
key was found, that redaction took place for that key. They didn't assert globally that when
running a full-fledged Spark app (whether or YARN or locally), that sensitive information
was not present in any of the logs or UI. Such a test would also prevent regressions from
happening in the future if someone unknowingly adds extra logging that publishes out sensitive
information to disk or UI.

Consequently, it was found that in some Java configurations, sensitive information was still
being leaked in the event logs under the {{SparkListenerEnvironmentUpdate}} event, like so:
"":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password

"secret_password" should have been redacted.

Moreover, previously redaction logic was only checking if the key matched the secret regex
pattern, it'd redact it's value. That worked for most cases. However, in the above case, the
key ( doesn't tell much, so the value needs to be searched. So the check
needs to be expanded to match against values as well.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message