airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Meickle <jmeic...@quantopian.com.INVALID>
Subject [2.0 spring cleaning] Require unique conn_id
Date Fri, 12 Apr 2019 14:51:10 GMT
Airflow fetches connections by name, but doesn't enforce unique names. My
team got bit by this, since it's very unexpected behavior for most types of
data entry. The reason for this behavior is explained in the docs:

"Many connections with the same conn_id can be defined and when that is the
case, and when the hooks uses the get_connection method from BaseHook,
Airflow will choose one connection randomly, allowing for some basic load
balancing and fault tolerance when used in conjunction with retries."

I think this is very non-intuitive UX. If we even want to support this
feature within Airflow - and I don't think that is a given - it would make
much more sense to require a unique (conn_id, conn_type) but allow storing
multiple related records. This wouldn't be a huge data modeling change, but
would require changing the web UI to appear as a form with subforms.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message