airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laura Lorenz <llor...@industrydive.com>
Subject Re: Airflow Testing Library
Date Fri, 05 May 2017 19:51:58 GMT
👍

On Fri, May 5, 2017 at 3:07 PM, Scott Halgrim <scott.halgrim@zapier.com>
wrote:

> Sounds A+ to me. By “both of you” did you include me? My first response
> was just to your email address.
>
> On May 5, 2017, 11:58 AM -0700, Sam Elamin <hussam.elamin@gmail.com>,
> wrote:
> > Ok sounds great folks
> >
> > Thanks for the detailed response laura! I'll invite both of you to the
> > group if you are happy and we can schedule a call for next week?
> >
> > How does that sound?
> > On Fri, 5 May 2017 at 17:41, Laura Lorenz <llorenz@industrydive.com>
> wrote:
> >
> > > We do! We developed our own little in-house DAG test framework which we
> > > could share insights on/would love to hear what other folks are up to.
> > > Basically we use mock a DAG's input data, use the BackfillJob API
> directly
> > > to call a DAG in a test, and compare its outputs to the intended result
> > > given the inputs. We use docker/docker-compose to manage services, and
> > > split our dev and test stack locally so that the tests have their own
> > > scheduler and metadata database and so that our CI tool knows how to
> > > construct the test stack as well.
> > >
> > > We co-opted the BackfillJob API for our own purposes here, but it
> seemed
> > > overly complicated and fragile to start and interact with our own
> > > in-test-process executor like we saw in a few of the tests in the
> Airflow
> > > test suite. So I'd be really interested on finding a way to streamline
> how
> > > to describe a test executor for both the Airflow test suite and
> people's
> > > own DAG testing and make that a first class type of API.
> > >
> > > Laura
> > >
> > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <hussam.elamin@gmail.com
> > > wrote:
> > >
> > > > Hi All
> > > >
> > > > A few people in the Spark community are interested in writing a
> testing
> > > > library for Airflow. We would love anyone who uses Airflow heavily in
> > > > production to be involved
> > > >
> > > > At the moment (AFAIK) testing your DAGs is a bit of a pain,
> especially if
> > > > you want to run them in a CI server
> > > >
> > > > Is anyone interested in being involved in the discussion?
> > > >
> > > > Kind Regards
> > > > Sam
> > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message