airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganesh Sivalingam (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1918) DockerOperator forcing Jinja templating on "command" when not required
Date Wed, 13 Dec 2017 13:35:00 GMT
Ganesh Sivalingam created AIRFLOW-1918:
------------------------------------------

             Summary: DockerOperator forcing Jinja templating on "command" when not required
                 Key: AIRFLOW-1918
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1918
             Project: Apache Airflow
          Issue Type: Bug
          Components: docker
    Affects Versions: Airflow 1.8
            Reporter: Ganesh Sivalingam


When creating a docker operator, and providing the command parameter like so:

{code: python}
create_cluster = DockerOperator(
    image="my/image",
    command="bash /home/ec2-user/create-cluster.sh",
    cpus=1
{code}

I get the following error:


{code:java}
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: Starting attempt 1 of 1
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
[2017-12-13 13:21:22,626] {base_task_runner.py:95} INFO - Subtask: 
[2017-12-13 13:21:22,639] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,639]
{models.py:1342} INFO - Executing <Task(DockerOperator): create_flintrock_cluster> on
2017-12-13 13:21:20.147853
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,652]
{models.py:1417} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent
call last):
[2017-12-13 13:21:22,653] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py",
line 1359, in run
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:     self.render_templates()
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py",
line 1579, in render_templates
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:     rendered_content =
rt(attr, content, jinja_context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py",
line 2245, in render_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:     return jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py",
line 812, in get_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:     return self._load_template(name,
self.make_globals(globals))
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py",
line 774, in _load_template
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:     cache_key = self.loader.get_source(self,
name)[1]
[2017-12-13 13:21:22,654] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py",
line 187, in get_source
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask:     raise TemplateNotFound(template)
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: TemplateNotFound: bash
/home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,655] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,653]
{models.py:1441} INFO - Marking task as FAILED.
[2017-12-13 13:21:22,667] {base_task_runner.py:95} INFO - Subtask: [2017-12-13 13:21:22,667]
{models.py:1462} ERROR - bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent
call last):
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/bin/airflow",
line 28, in <module>
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:     args.func(args)
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py",
line 422, in run
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:     pool=args.pool,
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py",
line 53, in wrapper
[2017-12-13 13:21:22,668] {base_task_runner.py:95} INFO - Subtask:     result = func(*args,
**kwargs)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py",
line 1359, in run
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:     self.render_templates()
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py",
line 1579, in render_templates
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:     rendered_content =
rt(attr, content, jinja_context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py",
line 2245, in render_template
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:     return jinja_env.get_template(content).render(**context)
[2017-12-13 13:21:22,669] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py",
line 812, in get_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:     return self._load_template(name,
self.make_globals(globals))
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py",
line 774, in _load_template
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:     cache_key = self.loader.get_source(self,
name)[1]
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py",
line 187, in get_source
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask:     raise TemplateNotFound(template)
[2017-12-13 13:21:22,670] {base_task_runner.py:95} INFO - Subtask: jinja2.exceptions.TemplateNotFound:
bash /home/ec2-user/create-cluster.sh
[2017-12-13 13:21:26,928] {jobs.py:2083} INFO - Task exited with return code 1
{code}

Within the DockerOperator code, if I comment out line 79 (`template_fields = ('command',)`)
then the task completes successfully.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message