airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramandumcs@gmail.com <ramandu...@gmail.com>
Subject Re: Upgrading Airflow in Prod
Date Wed, 19 Jun 2019 05:53:59 GMT
Thanks David & Teresa,
We are thinking of similar kind of approach where we would  have a dedicated Test env to test
the Airflow upgrades. We have to preserve the metadata so we will be upgrading the Airflow
version using Airflow cmd. Following steps would be done
-> Setup Mysql Slave to  Airflow's Mysql Metastore.
-> Stop/Kill Airflow scheduler & server
-> Detach Mysql Slave(It can be used for rollbacks)
-> Upgrade Airflow Server & Scheduler Image
-> Invoke airflow cli fro server image to upgrade DB
-> Start Airflow server and scheduler.
Restore metastore from slave If anything goes bad while upgrading Airflow cluster.


Thanks,
Raman Gupta

On 2019/06/12 17:39:11, David Klosowski <davidk@thinknear.com> wrote: 
> Hi Raman,
> 
> We do much the same that Teresa does.  We just have a staging and
> production instances and separated infrastructure environments for normal
> testing and deployments.  We do one of two things with upgrades
> 
> 1. Build out a separate environment just for upgrade testing (which doesn't
> take too long with Docker and cloud infrastructure)
> 2. Block our CI pipeline while validation of core features is done in
> staging
> 
> We also.
> 
> 1. Read the upgrade guide in the Airflow project
> 2. Review the JIRA tickets between versions
> 3. Assess any changes we may need to make to features like logging or our
> components (subdags, retry handlers, alerting, airflow variables, airflow
> configuration, UI etc)
> 4. Do end-to-end tests of our core custom operators and sensors verifying
> execution and retry cases
> 
> All-in-all I don't think I'd feel comfortable automating this yet b/c I
> think problems can exist in hidden places.
> 
> Cheers,
> David
> 
> On Wed, Jun 12, 2019 at 5:57 AM Teresa Martyny <
> teresa.martyny@omadahealth.com> wrote:
> 
> > Hi Raman,
> > We have a staging and preproduction instance. We block deploy of the docker
> > image to production on the successful full run of our core pipeline dags in
> > preproduction.
> >
> > If there's enough risk, we will disable the job that deploys to
> > preproduction, duplicate it, and deploy a branch to preproduction. Then we
> > can watch it run against the core pipeline and all external extracts for a
> > few runs to feel even more confident. At which point we will let it through
> > to master and our regular CI.
> >
> > Updating dependencies is the kind of scenario where we deploy a branch to
> > preproduction or staging depending on what the dependency is since the risk
> > is high. Another helpful thing is to wait a little and watch the issues in
> > Jira.
> >
> > Hope that helps,
> > Teresa
> >
> >
> > On Wed, Jun 12, 2019, 1:14 AM ramandumcs@gmail.com <ramandumcs@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > Are there any recommended steps/process to upgrade Airflow in production.
> > > We have a airflow setup in prod which is being actively used by different
> > > users. So ideally we would want to have a seamless upgrade but it seems
> > > Airflow upgrade are not backward compatible and might break few things.
> > > So what are the best practices for Airflow Upgrade in Production.
> > >
> > > Thanks,
> > > Raman Gupta
> > >
> >
> > --
> > This email may contain material that is confidential and/or privileged for
> > the sole use of the intended recipient. Any review, reliance, or
> > distribution by others or forwarding without express permission is
> > strictly
> > prohibited. If you are not the intended recipient, please contact the
> > sender and delete all copies. Also note that email is not an appropriate
> > way to send protected health information to Omada Health employees. Please
> > use your discretion when responding to this email.
> >
> 

Mime
View raw message