airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyrone Hinderson <thinder...@reonomy.com>
Subject Testing DAGs
Date Tue, 05 Jul 2016 18:30:03 GMT
What is a good example of a testing flow for new/updated DAGs? The "airflow
test" CLI command is great for testing individual tasks, but for testing
task interactions and the actual DAG topology (especially things like
branching or skipping) one needs to use the airflow scheduler. It seems to
me there are a few options here:

1. Start up an airflow scheduler locally, running with sqlite and
SequentialExecutor. Probably not production-like, but trivial to set up.

2. Deploy a more production-like airflow scheduler with the DAG you're
testing. Gives a better idea of how your code'll behave in production, but
feels like a pretty heavyweight testing solution.

3. Somehow submit your DAG-under-test to the production scheduler, watch it
run, edit and resubmit it if necessary, and remove it -- all without
disrupting the scheduler's other in-flight work. Gives the best preview of
your code in production, and low operational overhead for the dev.

#2 is mainly what I've been doing. #3 seems best, but I'm not immediately
aware of a clean way to add/update/remove DAGs in a remotely hosted
scheduler in an ad-hoc fashion?

Hopefully most of this makes sense, but please point out gaps in my
understanding!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message