airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Guziel <alex.guz...@airbnb.com.INVALID>
Subject Re: Runbook to upgrade Airflow
Date Wed, 04 Oct 2017 20:10:00 GMT
We pretty much just shut everything off (fence the db by changing the
password), upgrade the db, then turn everything back on and change the db
password back. We accept that there will be downtime for maintenance and do
it at a low impact time (like 20:00 utc ). Also, the thing with sqlalchemy
is a lot more things become breaking changes because of the way it writes
queries.

It might be good to have downtime even without schema changes to prevent
weird behavior if the changeset is large.

But for small stuff, depending on the scope of the change, workers don't
really need to be restarted, scheduler can be handled manually, and
webserver we do a rolling restart.

On Wed, Oct 4, 2017 at 7:54 AM Thoralf Gutierrez <thoralfgutierrez@gmail.com>
wrote:

> Thanks for your answer Alex.
>
> I guess you mean it won't work if there is a _breaking_ schema change
> right? But for new patch and minor versions (i.e. without breaking
> changes), I'm guessing that upgrading the db first should do the trick?
>
> @Airbnb team, how do you all do it? If I remember correctly, you often
> update your version of airflow to test out new releases. What does your
> runbook for Airflow upgrades look like?
>
> Cheers,
> Thoralf
>
> On Tue, 3 Oct 2017 at 13:04 Alex Guziel <alex.guziel@airbnb.com.invalid>
> wrote:
>
> > You won't be able to if there's a schema change.
> >
> > On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez <
> > thoralfgutierrez@gmail.com> wrote:
> >
> > > Hey everybody!
> > >
> > > Does anybody have some kind of runbook to upgrade airflow (with a
> Celery
> > > backend) without having any downtime (i.e. tasks keep on running as you
> > > upgrade)?
> > >
> > > I have this in mind, but not sure if I am missing something or if I
> > should
> > > be careful with the order of steps (especially for upgrading the db
> > > schema):
> > >
> > > 1. run airflow upgradedb from anywhere
> > >
> > > 2. one worker at a time
> > >   2a. make sure it doesn't start any new task.
> > >   2b. wait for all tasks to be finished
> > >   2c. run pip install airflow --upgrade
> > >   2d. re-enable worker
> > >
> > > 3. one webserver at a time
> > >   3a. kill webserver
> > >   3b. run pip install airflow --upgrade
> > >   3c. start webserver
> > >
> > > 4. scheduler
> > >   4a. kill scheduler
> > >   4b. run pip intsall airflow --upgrade
> > >   4c. start scheduler
> > >
> > > Thanks,
> > > Thoralf
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message