airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pala Muthiah <pala.muth...@airbnb.com.INVALID>
Subject Re: Meta DB transaction isolation level
Date Thu, 04 Jul 2019 06:10:22 GMT
We found at times some read queries lock up the task instance table, piling
up connections and  task heartbeat transactions, which takes longer to
recover. In READ COMMITTED, such long running and full table locks should
be avoided in many cases.

Basically we are trying to improve performance and recovery by reducing
locks.

If the ORM queries that Airflow ends up generating does not have
transactions that read same rows multiple times (within the transaction),
then consistency level is essentially the same.

Of course this is based on rough understanding and i am trying to verify if
others have actually implemented this.

On Tue, Jul 2, 2019 at 11:57 PM Ash Berlin-Taylor <ash@apache.org> wrote:

> Why do you want to change this? Is there something specific you are trying
> to achieve with changing this?
>
> (Periodic reminder: I personally recommend people use Postgres - I've
> always found it to be more predictable and stable than Mysql)
>
> -ash
>
> > On 3 Jul 2019, at 00:03, Pala Muthiah <pala.muthiah@airbnb.com.INVALID>
> wrote:
> >
> > Hello everyone
> >
> > Wanted to find out what is the transaction isolation level used in
> > airflow's MySQL meta db in your deployments.
> >
> > Does anybody use READ COMMITTED instead of the default REPEATABLE READ
> > transaction isolation level of MySQL? Did you observe any issues?
> >
> >
> > Thanks,
> > pala
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message