airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerard Toonstra (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-591) Datadog metrics service integration
Date Sun, 23 Oct 2016 21:41:58 GMT
Gerard Toonstra created AIRFLOW-591:
---------------------------------------

             Summary: Datadog metrics service integration
                 Key: AIRFLOW-591
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-591
             Project: Apache Airflow
          Issue Type: New Feature
          Components: contrib
            Reporter: Gerard Toonstra
            Priority: Minor


Datadog is an online service that collects metrics from servers and services. These metrics
can be anything quantifiable: the number of records inserted, cpu usage, amount of memory
left, number of logged in users.

I added a hook and a sensor to "contrib" which implements these data dog functions. The hook
can call 3 functions on the API; send_metric, query_metric and send_event. The sensor inspects
the event stream using a highly configurable filter and returns true or false if it has seen
it or not. 

The latter can be used for synchronization purposes with other schedulers or systems for example
and works similar to a message queue. 

The main purpose of the hook is to be able to record inserted records, count records with
history in an easy to filter format and configure alerts and monitors on datadog to keep track
of them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message