Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <1468000719847-f9ae8a50-980aa15e-98a6dd93@vo.yoo.ro>
References: <1468000719847-f9ae8a50-980aa15e-98a6dd93@vo.yoo.ro>
From: Maxime Beauchemin <maximebeauchemin@gmail.com>
Date: Sun, 10 Jul 2016 23:16:08 -0400
Message-ID: <CAHEEp7V8cthLchjT0M8PEoztem_OeDJZn5bTaf8sU9fM4DcX3Q@mail.gmail.com>
Subject: Re: Help reusing filepath from previous task
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113ec7fcd877c5053753920d
archived-at: Mon, 11 Jul 2016 03:16:16 -0000

--001a113ec7fcd877c5053753920d
Content-Type: text/plain; charset=UTF-8

You could:

copy_data = BashOperator(
    task_id='copy_data',
    bash_command='do_stuff.sh',
    params={'s3_path': s3_path}
    dag=dag)

then reference {{ params.s3_path }} as a templated element in do_stuff.sh

Max

On Fri, Jul 8, 2016 at 1:58 PM, <ziqr@vo.yoo.ro> wrote:

> Hello.
>
> I'd like to reuse some info given by a task to feed subsequent S3KeySensor
> checks, but after trying several ideas I'm not able to make this work.
>
> Basically my current DAG is as follow :
>
> copy_data = BashOperator(
>     task_id='copy_data',
>    # this script copies data to S3
>     bash_command='do_stuff.sh',
>     dag=dag)
>
> s3_path = xxxx
> for data_type in ("fileA", "fileB", "fileC"):
>     S3Sensor_task = S3KeySensor(
>         task_id='check_' + data_type,
>         poke_interval=20,
>         timeout=60,
>         retry_delay=timedelta(seconds=30),
>         bucket_key=s3_path +  data_type,
>         bucket_name='xxxx',
>         s3_conn_id='s3_conn_id',
>         dag=dag)
>     S3Sensor_task.set_upstream(copy_data)
>
> This works great, but I'd rather avoid duplicating the S3 path in the DAG
> and in the do_stuff.sh script.
> So I tought pushing S3 path from the do_stuff.sh to xcom variables, but
> then I need access to thoses items in my for loop.
> I've tried SubDag but once again I did not manage to read xcom.
>
> Could someone give me some hints ?
>
> Regards
>

--001a113ec7fcd877c5053753920d--