airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From purna pradeep <>
Subject S3keysonsor
Date Mon, 21 May 2018 17:16:08 GMT
Hi ,

I’m trying to evaluate airflow to see if it suits my needs.

Basically i can have below steps in a DAG

1)Look for a file arrival on given s3 location (this path is not completely
static) (i can use S3Keysensor in this step)

  i should be able to specify to look either for latest folder or 24hrs or
n number of days older folder which has _SUCCESS file as mentioned below

  sample file location(s):




2)invoke a simple restapi using HttpSimpleOperator once the above
dependency is met ,i can set upstream for step2 as step1

Does S3keysensor supports step1 out of the box?

Also in some cases i may to have a DAG without start date & end date it
just needs to be triggered once file is available in a given s3 location

 *Please suggest !*

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message