airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luke Diment <Luke.Dim...@westpac.co.nz>
Subject Re: How to know the DAG is starting to run
Date Fri, 11 May 2018 13:26:13 GMT
Airflow currently cannot be asked programmatically via an integration test if a dag has run
or what status a dag is at unless someone logs onto a box and runs a manual airflow command...

Sent from my iPhone

> On 12/05/2018, at 1:19 AM, Brian Greene <brian@heisenbergwoodworking.com> wrote:
>
> Okay I’ll bite...  WT* does that mean?
>
> one of the best things about airflow is how easy it is to connect disparate systems...
some would even say that’s much of the reason it exists..
>
> It records reasonable metadata into an rdms (I suppose you could argue for other designs,
but it’s pretty straightforward and for this workload easy enough to scale).
>
> So... what did you mean?
>
> Sent from a device with less than stellar autocorrect
>
>> On May 11, 2018, at 7:19 AM, Luke Diment <Luke.Diment@westpac.co.nz> wrote:
>>
>> We should really instrument an interface for airflow so integration is more succinct
between disparate systems!!!
>>
>> Sent from my iPhone
>>
>>> On 12/05/2018, at 12:09 AM, James Meickle <jmeickle@quantopian.com> wrote:
>>>
>>> Song:
>>>
>>> You can put an operator as the very first node in the DAG, and have
>>> everything else in the DAG depend on it. For example, this is the approach
>>> we use to only execute DAG tasks on stock market trading days.
>>>
>>> -James M.
>>>
>>>> On Fri, May 11, 2018 at 3:57 AM, Song Liu <songliu@outlook.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have something just want to be done only once when DAG is constructed,
>>>> but it seems that DAG will be instanced every time when run each of
>>>> operator.
>>>>
>>>> So is that there function in DAG that tell us it is starting to run now ?
>>>>
>>>> Thanks,
>>>> Song
>>>>
>>
>>
>>
>> The contents of this email and any attachments are confidential and may be legally
privileged. If you are not the intended recipient please advise the sender immediately and
delete the email and attachments. Any use, dissemination, reproduction or distribution of
this email and any attachments by anyone other than the intended recipient is prohibited.
>



The contents of this email and any attachments are confidential and may be legally privileged.
If you are not the intended recipient please advise the sender immediately and delete the
email and attachments. Any use, dissemination, reproduction or distribution of this email
and any attachments by anyone other than the intended recipient is prohibited.
Mime
View raw message