airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis O'Brien" <>
Subject best way to handle version upgrades of libraries used by tasks
Date Tue, 30 Jan 2018 17:13:47 GMT
Hi All,

I have a number of jobs that use scikit-learn for scoring players.
Occasionally I need to upgrade scikit-learn to take advantage of some new
features.  We have a single conda environment that specifies all the
dependencies for Airflow as well as for all of our DAGs.  So currently
upgrading scikit-learn means upgrading it for all DAGs that use it, and
retraining all models for that version.  It becomes a very involved task
and I'm hoping to find a better way.

One option is to use BashOperator (or something that wraps BashOperator)
and have bash use a specific conda environment with that version of
scikit-learn.  While simple, I don't like the idea of limiting task input
to the command line.  Still, an option.

Another option is the DockerOperator.  But when I asked around at a
previous Airflow Meetup, I couldn't find anyone actually using it.  It also
adds some complexity to the build and deploy process in that now I have to
maintain docker images for all my environments.  Still, not ruling it out.

And the last option I can think of is just heterogeneous workers.  We are
migrating our Airflow infrastructure to AWS ECS (from EC2) and plan on
having support for separate worker clusters, so this could include workers
with different conda environments.  I assume as long as a few key packages
are identical between scheduler and worker instances (airflow, redis,
celery?) the rest can be whatever.

Has anyone faced this problem and have some advice?  Am I missing any
simpler options?  Any thoughts much appreciated.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message