airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jeeyoung kim <jeeyou...@gmail.com>
Subject Recommended way of piping credentials / sensitive information on templated operators
Date Mon, 26 Feb 2018 19:22:47 GMT
Hi everyone,

I’m wondering how people work around accidentally writing credentials on
bash operator template page / logs.

For example, I may have PostgreSQL operator to copy data into Redshift.

COPY TABLE_NAME from 's3://.../something.manifest.json'
access_key_id '{{ params.AWS_ACCESS_KEY }}'
secret_access_key '{{ params.AWS_SECRET_KEY }}'

Or a command that exports from mongo

mongoexport \
  --assertExists \
  -h {{ connection.host }} \
  {% if connection.login %} -u {{ connection.login }} {% endif %}\
  {% if connection.get_password() %} -p {{ connection.get_password()
}} {% endif %}\
  -d {{ connection.schema }}
  ...

However, when this operator is executed (or when the template is rendered
on the UI), the credentials are written to the log files / clearly visible
on the UI, which is problematic.

There are many other cases where this can happen, and I’m wondering what is
a solution for it.

What would be ideal is:

   - Prevent credentials from accidentally being shown in “show rendered
   template” screen.
   - Prevent credentials from being written to the logs.

Thanks.

-Jeeyoung Kim
​

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message