airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Semet (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1847) Webhook Sensor
Date Sat, 25 Nov 2017 11:34:00 GMT
Semet created AIRFLOW-1847:

             Summary: Webhook Sensor
                 Key: AIRFLOW-1847
             Project: Apache Airflow
          Issue Type: Improvement
          Components: core, operators
            Reporter: Semet
            Priority: Minor
         Attachments: airflow-webhook-proposal.png

Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.

It is different than the {{dag_runs}} api in that the format is not airflow specific, it is
just a callback web url called by an external system on some even with its application specific
content. The content in really important and need to be sent to the dag (as XCom?)

Use Case:
- A Dag registers a WebHook sensor named {{<webhookname>}}
- An custom endpoint is exposed at {{http://myairflow.server/api/experimental/webhook/<webhookname>}}.
- I set this URL in the external system I wish to use the webhook from. Ex: github/gitlab
project webhook
- when the external application performs a request to this URL, this is automatically sent
to the WebHook sensor. For simplicity, we can have a JsonWebHookSensor that would be able
to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing it with the
json content as xcom.

If there are several requests at the same time, the system should be scalable enough to not
die or not slow down the webui. It is also possible to instantiate an independant flask/gunicorn
server to split the load. It would mean it runs on another port, but this could be just an
option in the configuration file or even a complete independant application ({{airflow webhookserver}}).
I saw recent changes integrated gunicorn in airflow core, guess it can help this use case.

This message was sent by Atlassian JIRA

View raw message