airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Praveenkumar Venkatesan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-19) How can I have an Operator B iterate over a list returned from upstream by Operator A?
Date Thu, 28 Apr 2016 21:36:12 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263070#comment-15263070
] 

Praveenkumar Venkatesan commented on AIRFLOW-19:
------------------------------------------------

yes. exactly. To definite it by tasks:

t1 - returns a list of paths to be copied over
t2 - Is trying to iterate over the list, (t3) create the disctp command, (t4) run the distCp
command, (t5) get the application id, (t6) monitor the state of the distcp job and (t7) complete.

> How can I have an Operator B iterate over a list returned from upstream by Operator A?
> --------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-19
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-19
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Praveenkumar Venkatesan
>            Priority: Minor
>              Labels: support
>
> Here is what I am trying to do exactly: https://gist.github.com/praveev/7b93b50746f8e965f7139ecba028490a
> the python operator log just returns the following
> [2016-04-28 11:56:22,296] {models.py:1041} INFO - Executing <Task(PythonOperator):
run_distcp_on_each> on 2016-04-28 11:56:12
> [2016-04-28 11:56:22,350] {python_operator.py:66} INFO - Done. Returned value was: None
> it didn't even print my kwargs and to_process data
> To simplify this. Lets say t1 returns 3 elements. I want to iterate over the list and
run t2 -> t3 for each element.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message