Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
From: Vincent Poulain <vincent.poulain@tinyclues.com>
Date: Fri, 24 Mar 2017 11:45:38 +0100
Message-ID: <CAGbaXZYGrQKdMeyZSBq7s52WZcxU5OLG3kUh9vfV=tLYe-LJQg@mail.gmail.com>
Subject: Dag & tasks testing
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a114cc192f6ba83054b77b28b
archived-at: Fri, 24 Mar 2017 10:46:06 -0000

--001a114cc192f6ba83054b77b28b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hello,

I have a quite important DAG (~20 tasks) using multiple Operators. I
dynamically create DAG using a dag =3D XXXDagFactory.create(**kwargs). I
would like to test this dag factory. For now I have a big method create. It
is something like

with Dag() as dag:
   t1 =3D PythonOperator(**kwargs)
   t2 =3D BashOperator(**kwargs)
   t3 =3D ECSOperator(**kwargs)
   ...
   t1 >> t2 >> t3
   ...
   return dag

With 300 lines of code :)

What do you recommend to write unit tests to assess that Operators are
called with right arguments and so on.

I don't want to have a unique test_create() method.
Should I create "business" method Operator wrapping technical Operator. For
instance :

class DoSomethindParticularOperator(PythonOperator):
    def __init__(**specific_args):

Or rather write just method returning task and test them.

def scoringTask(a, b, c):
    return ECSOperator(...)

Do you have some tips ?

Thanks!

--=20

*Vincent Poulain*

Senior Software Engineer


Office +33 1 75 50 67 26 <+33%201%2075%2050%2067%2026> | Mobile +33 6 21 82
87 62 | vincent@tinyclues.com <support@tinyclues.com>

Tinyclues | 51 rue =C3=89tienne Marcel, 75001 Paris

www.tinyclues.com <http://bit.ly/2hNL4Fs> | @tinyclues
<https://twitter.com/Tinyclues>

--001a114cc192f6ba83054b77b28b--