airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Kim (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-3001) Accumulative tis slow allocation of new schedule
Date Tue, 04 Sep 2018 08:35:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Kim updated AIRFLOW-3001:
-------------------------------
    Description: 
I have created very long term schedule in short interval (2~3 years as 10 min interval)

So, dag could be bigger and bigger as scheduling goes on.

Finally, at critical point (I don't know exactly when it is), the allocation of new task_instances
get slow and then almost stop.

I found that in this point, many slow query logs had occurred. (I was using mysql as meta
repository)

queries like this

"SELECT * FROM task_instance WHERE dag_id = '' and execution_date = ''"

I could resolve this issue by adding new index consist of dag_id and execution_date

So, I wanted 1.10 branch to be modified to create task_instance table with the index.

Thanks.

  was:
I have created very long term schedule in short interval (2~3 years as 10 min interval)

So, dag could be bigger and bigger as scheduling goes on.

Finally, at critical point (I don't know exactly when it is), the allocation of new task_instances
get slow and then almost stop.

I found that in this point, many slow query logs had occurred. (I was using mysql as meta
repository)

queries like this

"SELECT * FROM task_instance WHERE dag_id = '~' and execution_date = '~'"

I could resolve this issue by adding new index consist of dag_id and execution_date

So, I wanted 1.10 branch to be modified to create task_instance table with the index.

Thanks.


> Accumulative tis slow allocation of new schedule
> ------------------------------------------------
>
>                 Key: AIRFLOW-3001
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3001
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 1.10.0
>            Reporter: Jason Kim
>            Priority: Major
>
> I have created very long term schedule in short interval (2~3 years as 10 min interval)
> So, dag could be bigger and bigger as scheduling goes on.
> Finally, at critical point (I don't know exactly when it is), the allocation of new task_instances
get slow and then almost stop.
> I found that in this point, many slow query logs had occurred. (I was using mysql as
meta repository)
> queries like this
> "SELECT * FROM task_instance WHERE dag_id = '' and execution_date = ''"
> I could resolve this issue by adding new index consist of dag_id and execution_date
> So, I wanted 1.10 branch to be modified to create task_instance table with the index.
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message