Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_C5CFA8F6-CC62-48A8-9122-34B87789DFC9"
Mime-Version: 1.0 (Mac OS X Mail 11.0 \(3445.1.7\))
Subject: Re: Sensors
Date: Mon, 23 Oct 2017 19:10:42 +0200
References: <CACBF-_nuyjKvYkfF5cSTOMbc6Cwk=8pDzo8LBqzL6gAFu-=44A@mail.gmail.com>
 <CAOcXccHTa3qiqZhO0NKW=waFYT5LqDh_xacUZ9SyURqVAG7Pxg@mail.gmail.com>
To: dev@airflow.incubator.apache.org
In-Reply-To: <CAOcXccHTa3qiqZhO0NKW=waFYT5LqDh_xacUZ9SyURqVAG7Pxg@mail.gmail.com>
Message-Id: <FC54BC80-211D-4501-954D-F4F0F55AB37A@gmail.com>
archived-at: Mon, 23 Oct 2017 17:10:49 -0000

--Apple-Mail=_C5CFA8F6-CC62-48A8-9122-34B87789DFC9
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

I think you can do something like Azure functions blob storage binding =
and let that kick off a dag by triggering it from the Rest API:

=
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-=
storage-blob =
<https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings=
-storage-blob>

I don=E2=80=99t use Azure so it might not fit your case.

Bolke

> On 23 Oct 2017, at 16:15, Grant Nicholas =
<grantnicholas2015@u.northwestern.edu> wrote:
>=20
> It sounds like you want a background daemon that continuously monitors =
the
> status of some external system and triggers things on a condition. =
This
> does not sound like an ETL job, and thus airflow is not a great fit =
for
> this type of problem. That said, there are workarounds like you =
mentioned.
> One easy workaround if you can handle a delay between `condition =
happens ->
> dag triggers` is setting your controller dag to have a recurring =
schedule
> (ie: not None). Then when that controlling dag is triggered, you just
> perform your sensor check once and then trigger/don't trigger another =
dag
> depending on the condition. The thing I'd be worried about with your
> `trigger dagrun` approach is if the trigger dagrun operator fails for =
any
> reason you'll stop monitoring the external system, while with the =
scheduled
> approach you don't have to worry about the failure modes of retrying =
failed
> dags/etc.
>=20
> On Mon, Oct 23, 2017 at 2:30 AM, Niels Zeilemaker =
<niels@zeilemaker.nl>
> wrote:
>=20
>> Hi Guys,
>>=20
>> I've created a Sensor which is monitoring the number of files in an
>> Azure Blobstore. If the number of files increases, then I would like
>> to trigger another dag. This is more or less similar to the
>> example_trigger_controller_dag.py and example_trigger_target_dag.py
>> setup.
>>=20
>> However, after triggering the target DAG I would want my controller
>> DAG to start monitoring the Blobstore again. But since the schedule =
of
>> the controller DAG is set to None, it doesn't continue monitoring. I
>> "fixed" this by adding a TriggerDAG which schedules a new run of the
>> Controller DAG. But this feels a bit like a hack.
>>=20
>> Does someone have any experience which such a continuous monitoring
>> sensor? Or know of a better way to achieve this?
>>=20
>> Thanks,
>> Niels
>>=20


--Apple-Mail=_C5CFA8F6-CC62-48A8-9122-34B87789DFC9--