airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kaxil Naik (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-4724) Make params dict to be templated for operators
Date Wed, 07 Aug 2019 16:25:03 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kaxil Naik updated AIRFLOW-4724:
--------------------------------
    Fix Version/s:     (was: 1.10.4)
                   2.0.0

> Make params dict to be templated for operators
> ----------------------------------------------
>
>                 Key: AIRFLOW-4724
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4724
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: core, operators
>    Affects Versions: 1.10.3
>            Reporter: jack
>            Priority: Major
>             Fix For: 2.0.0
>
>
> Currently using params dict as:
> {code:java}
> EXEC_TIMESEPOCH =  "{{ execution_date.strftime('%s') }}"
> gcs_export_uri_template_filename = 'product_dwh-' + EXEC_TIMESEPOCH + '.csv'
> upload_file_ftp_op = BashOperator(
>    task_id='upload_file_ftp_task',
>    params={'filename':gcs_export_uri_template_filename},
>    bash_command="python3.6 /home/ubuntu/airflow/scripts/ranker.py  '{{ params.filename
}}'  " ,
>   dag=dag)
> {code}
> Gives:
> {code:java}
> python3.6 /home/ubuntu/airflow/scripts/ranker.py  'product_dwh-{{ execution_date.strftime('%s')
}}.csv'{code}
>  
> The BaseOperator says:
> {code:java}
> self.params = params or {} # Available in templates!{code}
> [https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py#L343]
> But as you can see above the code wasn't templated as expected.
>  
> I worked-around this by not using params dict as:
> {code:java}
> cmd = """python3.6 /home/ubuntu/airflow/scripts/ranker.py 'product_dwh-{{ execution_date.strftime('%s')
}}.csv' """
> upload_file_ftp_op = BashOperator(
>     task_id='upload_file_ftp_task',
>     bash_command = cmd,
>     dag=dag){code}
> This code works perfectly and while it's simpler and better the first code should have
still work.
>  
> A discussion about this has been on slack:
> [https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1559134166151100]
>  
> Since Slack doesn't save history forever [~feluelle] , [~dlamblin], if you have something
to comment please post it here so there will be a reference to whomever pick this one up.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message