airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Damian Momot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2175) Failed to upgradedb 1.8.2 -> 1.9.0
Date Tue, 06 Mar 2018 06:21:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387346#comment-16387346
] 

Damian Momot commented on AIRFLOW-2175:
---------------------------------------

This DAG is no longer in dag files but still exists in DB. For 99% it wasn't sub dag. It
was probably left in such strange state. As I remember this installation was initialized on
Airflow 1.8.1, then upgraded to 1.8.2.

Field is nullable in DB, DB is MySQL

 

 
{code:java}
DESCRIBE dag;
 
Field       Type            Null   Key Extra
...
'fileloc'   'varchar(2000)' 'YES'  ''  NULL ''
{code}
 

As a workaround I'll just manually alter record in DB, but null check is probably a good idea,
especially that field is nullable in DB - it might affect others

 

> Failed to upgradedb 1.8.2 -> 1.9.0
> ----------------------------------
>
>                 Key: AIRFLOW-2175
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2175
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db
>    Affects Versions: 1.9.0
>            Reporter: Damian Momot
>            Priority: Critical
>
> We've got airflow installation with hundreds of DAGs and thousands of tasks.
> During upgrade (1.8.2 -> 1.9.0) we've got following error.
> After analyzing stacktrace i've found that it's most likely caused by None value in 'fileloc'
field of Dag column. I checked database and indeed we've got one record with such value:
>  
>  
> {code:java}
> SELECT COUNT(*) FROM dag WHERE fileloc IS NULL;
> 1
> SELECT COUNT(*) FROM dag;
> 343
> {code}
>  
>  
> {code:java}
> Traceback (most recent call last):
>  File "/usr/local/bin/airflow", line 27, in <module>
>  args.func(args)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 913, in upgradedb
>  db_utils.upgradedb()
>  File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 320, in upgradedb
>  command.upgrade(config, 'heads')
>  File "/usr/local/lib/python2.7/dist-packages/alembic/command.py", line 174, in upgrade
>  script.run_env()
>  File "/usr/local/lib/python2.7/dist-packages/alembic/script/base.py", line 416, in run_env
>  util.load_python_file(self.dir, 'env.py')
>  File "/usr/local/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
>  module = load_module_py(module_id, path)
>  File "/usr/local/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py
>  mod = imp.load_source(module_id, path, fp)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/migrations/env.py", line 86, in
<module>
>  run_migrations_online()
>  File "/usr/local/lib/python2.7/dist-packages/airflow/migrations/env.py", line 81, in
run_migrations_online
>  context.run_migrations()
>  File "<string>", line 8, in run_migrations
>  File "/usr/local/lib/python2.7/dist-packages/alembic/runtime/environment.py", line 807,
in run_migrations
>  self.get_context().run_migrations(**kw)
>  File "/usr/local/lib/python2.7/dist-packages/alembic/runtime/migration.py", line 321,
in run_migrations
>  step.migration_fn(**kw)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py",
line 63, in upgrade
>  dag = dagbag.get_dag(ti.dag_id)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 232, in get_dag
>  filepath=orm_dag.fileloc, only_if_updated=False)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 249, in process_file
>  if not os.path.isfile(filepath):
>  File "/usr/lib/python2.7/genericpath.py", line 29, in isfile
>  st = os.stat(path)
> TypeError: coercing to Unicode: need string or buffer, NoneType found{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message