airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ash Berlin-Taylor (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-6265) Migration might behave wrongly when DAGS parsed use DB
Date Tue, 17 Dec 2019 22:00:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998626#comment-16998626
] 

Ash Berlin-Taylor commented on AIRFLOW-6265:
--------------------------------------------

I think we should silence any dag import errors that come up during running migrations --
even if it's an actual syntax error it's just going to be confusing to the user.

(Also we should probalby update our old migrations where we can to avoid parsing dags unless
absolutely necessary.)

> Migration might behave wrongly when DAGS parsed use DB
> ------------------------------------------------------
>
>                 Key: AIRFLOW-6265
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6265
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: models
>    Affects Versions: 1.10.2, 1.10.3, 1.10.4, 1.10.5, 1.10.6
>            Reporter: Jarek Potiuk
>            Priority: Minor
>
> When running migratedb there might be a problem displayed during migration - in case
some of the DAGs of the user use the DB in __init__ (for example reading connection table)
they might fail mid-migration with errors for example the one below.
>  
> The root cause is [https://github.com/apache/airflow/blob/master/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py#L70] -
DagBag object creation which parses all the DAGs.
>  
> It it is a minor annoyance - some of the TaskInstances can be not migrated properly with
the max_tries not copied from task to task_instance.
> airflow     | ========= ABOUT TO INIT DB ===========
> ...
> airflow     | DB: 
> [postgresql+psycopg2://airflow]
> :***@postgres:5432/airflow
> airflow     | [2019-12-17 18:17:06,471] \{db.py:350} INFO - Creating tables
> airflow     | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> airflow     | INFO  [alembic.runtime.migration] Will assume transactional DDL.
> airflow     | INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current
schema
> airflow     | INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f,
create is_encrypted
> airflow     | INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627,
maintain history for compatibility with earlier migrations
> ...
> airflow_db  | 2019-12-17 18:17:07.001 UTC [55] ERROR:  relation "connection" does not
exist at character 481
> airflow_db  | 2019-12-17 18:17:07.001 UTC [55] STATEMENT:  SELECT connection.password
AS connection_password, connection.extra AS connection_extra, connection.id AS connection_id,
connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host
AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login,
connection.port AS connection_port, connection.is_encrypted AS connection_is_encrypted, connection.is_extra_encrypted
AS connection_is_extra_encrypted
> airflow_db  | 	FROM connection
> airflow_db  | 	WHERE connection.conn_id = 'redshift_test_connection'
> airflow     | ERROR [airflow.models.DagBag] Failed to import: /airflow/repo/dags/papermill_test/test_papermill_connection_injection/papermill_dag.py
> airflow     | Traceback (most recent call last):
> airflow     |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py",
line 1236, in _execute_context
> airflow     |     cursor, statement, parameters, context
> airflow     |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py",
line 536, in do_execute
> airflow     |     cursor.execute(statement, parameters)
> airflow     | psycopg2.errors.UndefinedTable: relation "connection" does not exist
> airflow     | LINE 2: FROM connection
> ...
> airflow     | INFO  [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56,
Make xcom value column a large binary
> airflow     | INFO  [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff,
add ti job_id index
> airflow     | INFO  [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61,
Increase text size for MySQL (not relevant for other DBs' text types)
> airflow     | INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f,
Add time zone awareness
> airflow     | INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4,
kubernetes_resource_checkpointing
> airflow     | INFO  [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24,
kubernetes_resource_checkpointing
> ...
> ========= SLEEPING ===========



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message