airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brylie Christopher Oxley (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-3379) Support for AWS triggers (S3, SNS, SQS, CloudWatch, etc.)
Date Wed, 21 Nov 2018 08:45:00 GMT
Brylie Christopher Oxley created AIRFLOW-3379:

             Summary: Support for AWS triggers (S3, SNS, SQS, CloudWatch, etc.)
                 Key: AIRFLOW-3379
             Project: Apache Airflow
          Issue Type: Improvement
          Components: aws
            Reporter: Brylie Christopher Oxley

We would like to build a 'reactive', or event-driven, data pipeline. From what I can gather,
the primary Airflow DAG trigger is a timer (cron). However, it would be useful to trigger
dags on external events, namely AWS events like S3 file, SQS, SNS, and/or CloudWatch.

I note there is an experimental API, which could be triggered from an AWS Lambda, but would
add boilerplate and brittleness to the data pipeline.

What are our options for triggering Airflow DAGS from external AWS events?

This message was sent by Atlassian JIRA

View raw message