airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuliya Volkova (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2119) Celery worker fails when dag has space in filename
Date Tue, 20 Feb 2018 06:53:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369766#comment-16369766
] 

Yuliya Volkova commented on AIRFLOW-2119:
-----------------------------------------

[~paymahn], standard python import didn't support possible to load modules whitespace inside
names and to do this must be construction with {{ __import__("") but it's anyway terrible,
I don't think what need to support space inside file names? why you need it? }}

> Celery worker fails when dag has space in filename
> --------------------------------------------------
>
>                 Key: AIRFLOW-2119
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2119
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: celery, worker
>    Affects Versions: 1.9.0
>            Reporter: Paymahn Moghadasian
>            Priority: Minor
>
> A dag whose filename has a space will cause celery workers to fail as follows:
>  
> {noformat}
> [2018-02-16 22:58:55,976] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
> [2018-02-16 22:58:56,021] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
> [2018-02-16 22:58:56,322] {configuration.py:206} WARNING - section/key [celery/celery_ssl_active]
not found in config
> [2018-02-16 22:58:56,322] {default_celery.py:41} WARNING - Celery Executor will run without
SSL
> [2018-02-16 22:58:56,323] {__init__.py:45} INFO - Using executor CeleryExecutor
> Starting flask
> [2018-02-16 22:58:56,403] {_internal.py:88} INFO -  * Running on http://0.0.0.0:8793/
(Press CTRL+C to quit)
> [2018-02-16 22:59:25,181] {celery_executor.py:50} INFO - Executing command in Celery:
airflow run broken_hello_world dummy_task 2018-02-15T12:00:00 --local -sd /home/paymahn/scheduler/airflow-home/dags/hello
world.py
> [2018-02-16 22:59:25,569] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
> [2018-02-16 22:59:25,610] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
> [2018-02-16 22:59:25,885] {configuration.py:206} WARNING - section/key [celery/celery_ssl_active]
not found in config
> [2018-02-16 22:59:25,885] {default_celery.py:41} WARNING - Celery Executor will run without
SSL
> [2018-02-16 22:59:25,886] {__init__.py:45} INFO - Using executor CeleryExecutor
> usage: airflow [-h]
>                {flower,kerberos,upgradedb,worker,render,serve_logs,backfill,task_state,dag_state,test,connections,pause,unpause,list_tasks,scheduler,run,list_dags,webserver,trigger_dag,version,pool,resetdb,clear,variables,initdb,task_failed_deps}
>                ...
> airflow: error: unrecognized arguments: world.py
> [2018-02-16 22:59:26,055] {celery_executor.py:54} ERROR - Command 'airflow run broken_hello_world
dummy_task 2018-02-15T12:00:00 --local -sd /home/paymahn/scheduler/airflow-home/dags/hello
world.py' returned non-zero exit status 2
> [2018-02-16 22:59:26,117: ERROR/ForkPoolWorker-16] Task airflow.executors.celery_executor.execute_command[114e9ff2-380e-4ba6-84e4-c913ea85189c]
raised unexpected: AirflowException('Celery command failed',)
> Traceback (most recent call last):
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
line 52, in execute_command
>     subprocess.check_call(command, shell=True)
>   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command 'airflow run broken_hello_world dummy_task 2018-02-15T12:00:00
--local -sd /home/paymahn/scheduler/airflow-home/dags/hello world.py' returned non-zero exit
status 2
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
line 374, in trace_task
>     R = retval = fun(*args, **kwargs)
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
line 629, in __protected_call__
>     return self.run(*args, **kwargs)
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
line 55, in execute_command
>     raise AirflowException('Celery command failed')
> airflow.exceptions.AirflowException: Celery command failed
> [2018-02-16 22:59:27,420] {celery_executor.py:50} INFO - Executing command in Celery:
airflow run broken_hello_world dummy_task 2018-02-16T22:59:26.096432 --local -sd /home/paymahn/scheduler/airflow-home/dags/hello
world.py
> [2018-02-16 22:59:27,840] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
> [2018-02-16 22:59:27,877] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
> [2018-02-16 22:59:28,148] {configuration.py:206} WARNING - section/key [celery/celery_ssl_active]
not found in config
> [2018-02-16 22:59:28,148] {default_celery.py:41} WARNING - Celery Executor will run without
SSL
> [2018-02-16 22:59:28,149] {__init__.py:45} INFO - Using executor CeleryExecutor
> usage: airflow [-h]
>                {resetdb,version,dag_state,trigger_dag,connections,task_state,variables,upgradedb,webserver,kerberos,pool,serve_logs,run,list_dags,scheduler,render,flower,task_failed_deps,worker,unpause,backfill,test,initdb,list_tasks,pause,clear}
>                ...
> airflow: error: unrecognized arguments: world.py
> [2018-02-16 22:59:28,316] {celery_executor.py:54} ERROR - Command 'airflow run broken_hello_world
dummy_task 2018-02-16T22:59:26.096432 --local -sd /home/paymahn/scheduler/airflow-home/dags/hello
world.py' returned non-zero exit status 2
> [2018-02-16 22:59:28,370: ERROR/ForkPoolWorker-8] Task airflow.executors.celery_executor.execute_command[5bd0d338-1df5-4ced-8a29-f87d68c643bd]
raised unexpected: AirflowException('Celery command failed',)
> Traceback (most recent call last):
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
line 52, in execute_command
>     subprocess.check_call(command, shell=True)
>   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command 'airflow run broken_hello_world dummy_task 2018-02-16T22:59:26.096432
--local -sd /home/paymahn/scheduler/airflow-home/dags/hello world.py' returned non-zero exit
status 2
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
line 374, in trace_task
>     R = retval = fun(*args, **kwargs)
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/celery/app/trace.py",
line 629, in __protected_call__
>     return self.run(*args, **kwargs)
>   File "/home/paymahn/scheduler/venv/lib/python3.5/site-packages/airflow/executors/celery_executor.py",
line 55, in execute_command
>     raise AirflowException('Celery command failed')
> airflow.exceptions.AirflowException: Celery command failed
> {noformat}
> I suspect the problem is that there's an unescaped space in the name of the file that
gets passed to one of the subprocess calls.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message