airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mariusz Górski (Jira) <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-5774) PapermillOperator fails to render parameters
Date Thu, 31 Oct 2019 08:49:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963774#comment-16963774
] 

Mariusz Górski commented on AIRFLOW-5774:
-----------------------------------------

[~playermanny2] updated the link, it's already on master but not a part of latest release. 
Please see: [https://github.com/apache/airflow/commit/a6daeb544e815fe350a96d24ae3bb14aee4079a7#diff-5fe5f93b69bf5129e9c10232ee13be30]

> PapermillOperator fails to render parameters
> --------------------------------------------
>
>                 Key: AIRFLOW-5774
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5774
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: utils
>    Affects Versions: 1.10.4
>            Reporter: Mariusz Górski
>            Priority: Minor
>              Labels: bug
>
> *Problem*
> When running sample PapermillOperator described in docs, it fails with
> {code:java}
> //[2019-10-27 10:48:53,134] {{taskinstance.py:1047}} ERROR - Can't compile non template
nodes
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
922, in _run_raw_task
>     result = task_copy.execute(context=context)
>   File "/usr/local/lib/python3.7/site-packages/airflow/operators/papermill_operator.py",
line 71, in execute
>     logging.info(inlet.as_dict())
>   File "/usr/local/lib/python3.7/site-packages/airflow/lineage/datasets.py", line 85,
in as_dict
>     attributes[key] = env.from_string(value).render(**self.context)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 880, in from_string
>     return cls.from_code(self, self.compile(source), globals, None)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 581, in compile
>     defer_init=defer_init)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 543, in _generate
>     optimized=self.optimized)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/compiler.py", line 78, in generate
>     raise TypeError('Can\'t compile non template nodes')
> TypeError: Can't compile non template nodes
> {code}
> *Reason*
> It's because: _class NoteBook(DataSet)_ expects _parameters_ to be _strings_, while
papermill _execute_notebook_ function requires _parameters_ to be _dicts_. When we pass
dict to _NoteBook.parameters_, then jinja templating fails when dict is passed to _env.from_string(value).render(**context)_ call. 
> *Solution:*
> Handling of nested structs in DataSet class
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message