airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cieplucha, Michal" <michal.cieplu...@intel.com>
Subject too many long running tasks in sensors
Date Wed, 18 Oct 2017 12:58:30 GMT
Hello,

We are using sensors for lightweight but taking long time tasks. It's like monitoring results
of some testing executed on remote machine. We would like to see such a task in a DAG but
with so much tasks/dags in running state we will hit max number of processes/dag runs (defined
in airflow.cfg) soon. Is it possible to have a sensor, which instead of sleep and keep the
process would just exit and be rescheduled later? Are we using sensors in wrong manner?

Thanks
mC

---------------------------------------------------------------------------------------------------------------------------------
I am an Intel employee. All comments and opinions are my own and do not represent the views
of Intel.
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy
Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac
informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie
nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the
intended recipient(s). If you are not the intended recipient, please contact the sender and
delete all copies; any review or distribution by
others is strictly prohibited.


Mime
View raw message