airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasad Bhalerao <prasadbhalerao1...@gmail.com>
Subject Re: Capture DAG states
Date Fri, 11 Jun 2021 17:58:10 GMT
Hi,

It means I have to call these apis explicitly right!
But that is not my requirement, I want to fire my custom events when DAG
state changes.

As per my initial analysis I could think of using on_failure_callback
and on_success_callback
while creating DAGs. These callbacks will trigger my custom events. What I
understand is that these callbacks will be executed on the scheduler node.
So I have to use some mechanism like rest end pt or web hooks to post these
events to my eventing module.
Does this callback provide DAG_ID in context object?

Please correct me if I am wrong.

Thanks,
Prasad

On Fri, Jun 11, 2021 at 10:58 PM Bruno Gonzalez <bruno@homelight.com> wrote:

> Prasad, if you are using Airflow 2 you can use the new stable API. The
> state of the DAG will be a combination of List DAG runs
> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dag_runs>
to
> get the *dag_run_id* and Get a DAG run
> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dag_run>
to
> get the *state*, that is the one that you're looking for.
>
>  Hope this helps.
>
> On Fri, Jun 11, 2021 at 11:50 AM Prasad Bhalerao <
> prasadbhalerao1983@gmail.com> wrote:
>
>> Let me rephrase my question.
>>
>> Is there anyway to capture DAG states?
>>
>> Based on these state I want to trigger the events.
>>
>>
>>
>> On Fri, 11 Jun 2021 at 8:12 PM, Ash Berlin-Taylor <ash@apache.org> wrote:
>>
>>> What do you mean by "halted"? That isn't the name of any state in
>>> Airflow.
>>>
>>> -ash
>>>
>>> On Fri, Jun 11 2021 at 18:50:51 +0530, Prasad Bhalerao <
>>> prasadbhalerao1983@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I want to capture the following events. I want to capture these states
>>> and generate an event for each.
>>> As part of this I want to capture DAG ID and DAG Name etc.
>>>
>>> 1) DAG\pipeline started
>>> 2) DAG\pipeline halted
>>> 3) DAG\pipeline failed
>>> 4) DAG\pipeline success.
>>>
>>> Can someone please help me with this?
>>>
>>>
>>> Thanks,
>>> Prasad
>>>
>>>

Mime
View raw message