airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Bhat <vijaysb...@gmail.com>
Subject Git webhooks with Airflow
Date Wed, 07 Sep 2016 22:25:21 GMT
Hi all,

First off, I want to thank the Airflow community for developing a fantastic
data pipelining platform. I used Dataswarm extensively while I was at
Facebook, and it's awesome to see most of the functionality available for
the rest of the world to use in the form of Airflow.

What I haven't found in the documentation is a prescribed way to connect
the source control repo for the DAG code to the Airflow DAG folder to make
sure the latest code changes are picked up by the scheduler. In the Airflow
forums, I have seen people mention using cron / chef / puppet etc, but no
git webhook (https://developer.github.com/v3/repos/hooks/) based methods.

Using webhooks would prevent the need to poll the repo for changes. For
example, Jenkins uses webhooks to auto-trigger builds -
https://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin#GithubPlugin-TriggerabuildwhenachangeispushedtoGitHub.
Does Airflow have a way of configuring something similar?

Thanks!
Vijay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message