airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franco Peschiera <franco.peschi...@gmail.com>
Subject Re: Stopping / killing dag runs gracefully
Date Fri, 22 Jan 2021 10:23:45 GMT
Thanks for the answer Anton.
It definitely makes sense. In fact, that's what we already do: we always
pass a time limit to the optimization task and it's respected. But we still
want to reserve the possibility to stop the run before the time limit has
been reached. Tasks can have a time limit from several seconds to several
hours (or days) and some example reasons why we may want to interrupt an
ongoing task before the time limit are:

* the user realizes the input data is wrong, and so it doesn't make sense
to keep running the task.
* the log from the optimization task process shows that the task is not
going according to plan, and so the user wants to interrupt it to check why
and send another one.
* some input data or hypothesis has changed and the ongoing optimization
task is obsolete.

thanks again!

Franco


On Fri, Jan 22, 2021 at 10:29 AM Anton Erholt <
anton.erholt@epidemicsound.com> wrote:

> Apologies for not answering your question about how to stop DAG runs. I do
> not know that. However, I wonder if it would make sense to pass along how
> long the job should as a parameter to the optimization task, and when it
> times out, exit/return appopriately so Airflow can read it?
>
> Best,
> Anton
>
> Den fre 22 jan. 2021 08:01Franco Peschiera <franco.peschiera@gmail.com>
> skrev:
>
>> Hello again everyone,
>>
>> As the title says: I would like to be able to stop / kill a dagrun. I’ve
>> seen this question asked here
>> <https://stackoverflow.com/questions/43631693/how-to-stop-kill-airflow-tasks-from-the-ui>
>> and here
>> <https://stackoverflow.com/questions/49039386/how-do-i-stop-an-airflow-dag>
>> .
>> Several solutions are proposed but I was wondering if there is a
>> “correct” way to stop a dagrun. I’m guessing from the airflow 2.0 docs of
>> the REST API that it is probably not possible from there (since I did not
>> see that). And since I do not see it anywhere in the docs, I fear there may
>> not be a good way to do that properly. Is that so? Is there even an
>> “improper” way?
>>
>> As context, the tasks that we want to schedule are optimization tasks
>> that do not have a fixed time to run. Users usually put a time limit, e.g.,
>> an hour, and we would anyway put one by default if they don’t. But, in
>> general, users may want to stop an execution if they see it takes too long
>> or if they want to change something before running it again. So scheduling
>> and stopping dagruns should be a “common” thing to do.
>>
>> Thanks as always!
>>
>> Franco
>>
>

Mime
View raw message