airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-2215) celery task launches subprocess without environment vars
Date Tue, 03 Apr 2018 05:45:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423503#comment-16423503
] 

ASF subversion and git services commented on AIRFLOW-2215:
----------------------------------------------------------

Commit d1f94fe20ea47f18f353ed47a07bc589a4a9c063 in incubator-airflow's branch refs/heads/master
from [~johnarnold]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d1f94fe ]

[AIRFLOW-2215] Pass environment to subproces.Popen in base_task_runner

Closes #3183 from johnarnold/env2


> celery task launches subprocess without environment vars
> --------------------------------------------------------
>
>                 Key: AIRFLOW-2215
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2215
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler, worker
>    Affects Versions: 1.9.0
>            Reporter: John Arnold
>            Priority: Major
>             Fix For: 2.0.0
>
>
> The celery task that is ran by CeleryExecutor does not preserve the environment variables
when launching task instances in a subprocess.  specifically, AIRFLOW_HOME is lost.  In
my environment, this results in the command failing.  Typically the command is something
like 'airflow run <dag> <operator> <timestamp> -sd < dag path>'
> @app.task
> def execute_command(command):
>  log = LoggingMixin().log
>  log.info("Executing command in Celery: %s", command)
>  env = os.environ.copy()
>  try:
>  subprocess.check_call(command, shell=True, close_fds=True, env=env)
>  except subprocess.CalledProcessError as e:
>  log.error(e)
>  raise AirflowException('Celery command failed')
>  
> I believe the environment vars should be preserved in the subtask.  Also, the logging
kind of sucks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message