airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-6350) security - spark submit operator logging+exceptions should mask passwords
Date Fri, 27 Dec 2019 12:56:01 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-6350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004168#comment-17004168
] 

ASF subversion and git services commented on AIRFLOW-6350:
----------------------------------------------------------

Commit d3988a03e11c715701bf72bf250ef7dd91d17a59 in airflow's branch refs/heads/v1-10-test
from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d3988a0 ]

[AIRFLOW-6350] security - spark submit operator logging+exceptions sh… (#6917)

* [AIRFLOW-6350] security - spark submit operator logging+exceptions should mask passwords

(cherry picked from commit 468814802e3126c3cd48ef31a3287f3fd041498b)


> security - spark submit operator logging+exceptions should mask passwords 
> --------------------------------------------------------------------------
>
>                 Key: AIRFLOW-6350
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6350
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks, operators
>    Affects Versions: 1.10.3
>            Reporter: t oo
>            Assignee: t oo
>            Priority: Major
>             Fix For: 2.0.0
>
>
> contrib/hooks/spark_submit_hook.py    
> Mask passwords in spark submit cmd AND error stacktrace
>  
> *add*
> def _mask_cmd(self, connection_cmd):
>  # Mask any password related fields in application args with key value pair where key
contains password (case insensitive), e.g. HivePassword='abc'
> connection_cmd_masked = re.sub(r"(\S*?(?:secret|password)\S*?\s*=\s*')[^']*(?=')", r'\1******',
' '.join(connection_cmd), flags=re.I)
> return connection_cmd_masked
>  
> *BEFORE*
> self.log.info("Spark-Submit cmd: %s", connection_cmd)
>  
> *AFTER*
> self.log.info("Spark-Submit cmd: %s", self._mask_cmd(connection_cmd))
>  
> *BEFORE*
>  if returncode or (self._is_kubernetes and self._spark_exit_code != 0):
> raise AirflowException(
> "Cannot execute: {}. Error code is: {}.".format(
> spark_submit_cmd, returncode
> )
> )
>  
> *AFTER*
> if returncode or (self._is_kubernetes and self._spark_exit_code != 0):
> raise AirflowException(
> "Cannot execute: {}. Error code is: {}.".format(
> self._mask_cmd(spark_submit_cmd), returncode
> )
> )



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message