airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fokko Driesprong (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-2215) celery task launches subprocess without environment vars
Date Thu, 15 Mar 2018 08:07:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Fokko Driesprong resolved AIRFLOW-2215.
---------------------------------------
       Resolution: Fixed
    Fix Version/s: 2.0.0

Issue resolved by pull request #3126
[https://github.com/apache/incubator-airflow/pull/3126]

> celery task launches subprocess without environment vars
> --------------------------------------------------------
>
>                 Key: AIRFLOW-2215
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, worker
>    Affects Versions: 1.9.0
>            Reporter: John Arnold
>            Priority: Major
>             Fix For: 2.0.0
>
>
> The celery task that is ran by CeleryExecutor does not preserve the environment variables
when launching task instances in a subprocess.  specifically, AIRFLOW_HOME is lost.  In
my environment, this results in the command failing.  Typically the command is something
like 'airflow run <dag> <operator> <timestamp> -sd < dag path>'
> @app.task
> def execute_command(command):
>  log = LoggingMixin().log
>  log.info("Executing command in Celery: %s", command)
>  env = os.environ.copy()
>  try:
>  subprocess.check_call(command, shell=True, close_fds=True, env=env)
>  except subprocess.CalledProcessError as e:
>  log.error(e)
>  raise AirflowException('Celery command failed')
>  
> I believe the environment vars should be preserved in the subtask.  Also, the logging
kind of sucks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message