Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAHogpC5jf4nb+9+GaGF5FY9kFxhYYsk-z64s6t1kxUouCkoKUA@mail.gmail.com>
References: <CAHogpC5jf4nb+9+GaGF5FY9kFxhYYsk-z64s6t1kxUouCkoKUA@mail.gmail.com>
From: Laura Lorenz <llorenz@industrydive.com>
Date: Fri, 5 May 2017 12:40:59 -0400
Message-ID: <CAJQxwiGuPPGX+RYQc8jcD0sHxBPs7XVTV_ewV+pyH4oyg3R5OA@mail.gmail.com>
Subject: Re: Airflow Testing Library
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1142c9804f2a0d054ec99080
archived-at: Fri, 05 May 2017 16:41:51 -0000

--001a1142c9804f2a0d054ec99080
Content-Type: text/plain; charset=UTF-8

We do! We developed our own little in-house DAG test framework which we
could share insights on/would love to hear what other folks are up to.
Basically we use mock a DAG's input data, use the BackfillJob API directly
to call a DAG in a test, and compare its outputs to the intended result
given the inputs. We use docker/docker-compose to manage services, and
split our dev and test stack locally so that the tests have their own
scheduler and metadata database and so that our CI tool knows how to
construct the test stack as well.

We co-opted the BackfillJob API for our own purposes here, but it seemed
overly complicated and fragile to start and interact with our own
in-test-process executor like we saw in a few of the tests in the Airflow
test suite. So I'd be really interested on finding a way to streamline how
to describe a test executor for both the Airflow test suite and people's
own DAG testing and make that a first class type of API.

Laura

On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <hussam.elamin@gmail.com> wrote:

> Hi All
>
> A few people in the Spark community are interested in writing a testing
> library for Airflow. We would love anyone who uses Airflow heavily in
> production to be involved
>
> At the moment (AFAIK) testing your DAGs is a bit of a pain, especially if
> you want to run them in a CI server
>
> Is anyone interested in being involved in the discussion?
>
> Kind Regards
> Sam
>

--001a1142c9804f2a0d054ec99080--