airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Poulain <vincent.poul...@tinyclues.com>
Subject Dag & tasks testing
Date Fri, 24 Mar 2017 10:45:38 GMT
Hello,

I have a quite important DAG (~20 tasks) using multiple Operators. I
dynamically create DAG using a dag = XXXDagFactory.create(**kwargs). I
would like to test this dag factory. For now I have a big method create. It
is something like

with Dag() as dag:
   t1 = PythonOperator(**kwargs)
   t2 = BashOperator(**kwargs)
   t3 = ECSOperator(**kwargs)
   ...
   t1 >> t2 >> t3
   ...
   return dag

With 300 lines of code :)

What do you recommend to write unit tests to assess that Operators are
called with right arguments and so on.

I don't want to have a unique test_create() method.
Should I create "business" method Operator wrapping technical Operator. For
instance :

class DoSomethindParticularOperator(PythonOperator):
    def __init__(**specific_args):

Or rather write just method returning task and test them.

def scoringTask(a, b, c):
    return ECSOperator(...)

Do you have some tips ?

Thanks!

-- 

*Vincent Poulain*

Senior Software Engineer



Office +33 1 75 50 67 26 <+33%201%2075%2050%2067%2026> | Mobile +33 6 21 82
87 62 | vincent@tinyclues.com <support@tinyclues.com>

Tinyclues | 51 rue √Čtienne Marcel, 75001 Paris

www.tinyclues.com <http://bit.ly/2hNL4Fs> | @tinyclues
<https://twitter.com/Tinyclues>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message