airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arnie Salazar <asala...@riotgames.com>
Subject Re: Airflow Testing Library
Date Mon, 08 May 2017 23:14:12 GMT
Scratch that. I see the whole video now.

On Mon, May 8, 2017 at 3:33 PM Arnie Salazar <asalazar@riotgames.com> wrote:

> Thanks Sam!
>
> Is there a part 2 to the video? If not, can you post the "next steps"
> notes you took whenever you have a chance?
>
> Cheers,
> Arnie
>
> On Mon, May 8, 2017 at 3:08 PM Sam Elamin <hussam.elamin@gmail.com> wrote:
>
>> Hi Folks
>>
>> For those of you who missed it, you can catch the discussion from the link
>> on this tweet <https://twitter.com/samelamin/status/861703888298225670>
>>
>> Please do share and feel free to get involved as the more feedback we get
>> the better the library we create is :)
>>
>> Regards
>> Sam
>>
>> On Mon, May 8, 2017 at 9:43 PM, Sam Elamin <hussam.elamin@gmail.com>
>> wrote:
>>
>> > Bit late notice but the call is happening today at 9 15 utc so in about
>> >  30 mins or so
>> >
>> > It will be recorded but if anyone would like to join in on the
>> discussion
>> > the hangout link is https://hangouts.google.com/hangouts/_/
>> > mbkr6xassnahjjonpuvrirxbnae
>> >
>> > Regards
>> > Sam
>> >
>> > On Fri, 5 May 2017 at 21:35, Ali Uz <aliuz1@gmail.com> wrote:
>> >
>> >> I am also very interested in seeing how this turns out. Even though we
>> >> don't have a testing framework in-place on the project I am working
>> on, I
>> >> would very much like to contribute to some general framework for
>> testing
>> >> DAGs.
>> >>
>> >> As of now we are just implementing dummy tasks that test our actual
>> tasks
>> >> and verify if the given input produces the expected output. Nothing
>> crazy
>> >> and certainly not flexible in the long run.
>> >>
>> >>
>> >> On Fri, 5 May 2017 at 22:59, Sam Elamin <hussam.elamin@gmail.com>
>> wrote:
>> >>
>> >> > Haha yes Scott you are in!
>> >> > On Fri, 5 May 2017 at 20:07, Scott Halgrim <scott.halgrim@zapier.com
>> >
>> >> > wrote:
>> >> >
>> >> > > Sounds A+ to me. By “both of you” did you include me? My first
>> >> response
>> >> > > was just to your email address.
>> >> > >
>> >> > > On May 5, 2017, 11:58 AM -0700, Sam Elamin <
>> hussam.elamin@gmail.com>,
>> >> > > wrote:
>> >> > > > Ok sounds great folks
>> >> > > >
>> >> > > > Thanks for the detailed response laura! I'll invite both
of you
>> to
>> >> the
>> >> > > > group if you are happy and we can schedule a call for next
week?
>> >> > > >
>> >> > > > How does that sound?
>> >> > > > On Fri, 5 May 2017 at 17:41, Laura Lorenz <
>> llorenz@industrydive.com
>> >> >
>> >> > > wrote:
>> >> > > >
>> >> > > > > We do! We developed our own little in-house DAG test
framework
>> >> which
>> >> > we
>> >> > > > > could share insights on/would love to hear what other
folks
>> are up
>> >> > to.
>> >> > > > > Basically we use mock a DAG's input data, use the BackfillJob
>> API
>> >> > > directly
>> >> > > > > to call a DAG in a test, and compare its outputs to
the
>> intended
>> >> > result
>> >> > > > > given the inputs. We use docker/docker-compose to manage
>> services,
>> >> > and
>> >> > > > > split our dev and test stack locally so that the tests
have
>> their
>> >> own
>> >> > > > > scheduler and metadata database and so that our CI tool
knows
>> how
>> >> to
>> >> > > > > construct the test stack as well.
>> >> > > > >
>> >> > > > > We co-opted the BackfillJob API for our own purposes
here, but
>> it
>> >> > > seemed
>> >> > > > > overly complicated and fragile to start and interact
with our
>> own
>> >> > > > > in-test-process executor like we saw in a few of the
tests in
>> the
>> >> > > Airflow
>> >> > > > > test suite. So I'd be really interested on finding a
way to
>> >> > streamline
>> >> > > how
>> >> > > > > to describe a test executor for both the Airflow test
suite and
>> >> > > people's
>> >> > > > > own DAG testing and make that a first class type of
API.
>> >> > > > >
>> >> > > > > Laura
>> >> > > > >
>> >> > > > > On Fri, May 5, 2017 at 11:46 AM, Sam Elamin <
>> >> hussam.elamin@gmail.com
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Hi All
>> >> > > > > >
>> >> > > > > > A few people in the Spark community are interested
in
>> writing a
>> >> > > testing
>> >> > > > > > library for Airflow. We would love anyone who uses
Airflow
>> >> heavily
>> >> > in
>> >> > > > > > production to be involved
>> >> > > > > >
>> >> > > > > > At the moment (AFAIK) testing your DAGs is a bit
of a pain,
>> >> > > especially if
>> >> > > > > > you want to run them in a CI server
>> >> > > > > >
>> >> > > > > > Is anyone interested in being involved in the discussion?
>> >> > > > > >
>> >> > > > > > Kind Regards
>> >> > > > > > Sam
>> >> > > > > >
>> >> > > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message