airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Gupta <>
Subject Fusing operators together
Date Wed, 14 Nov 2018 20:56:07 GMT
*[Please let me know if this is NOT the correct place for such a query]*

Hello maintainers and committers,
I've stumbled upon this design decision for my Airflow project. Any
pointers would be helpful.


   - I'm in the process of deploying Airflow and I've felt the need to
   merge groups of operators that form a single logical task (to clear the
   clutter in huge DAGs)
   - The most common use-case would be coupling an operator and the
   corresponding sensor. For instance, one might want to chain together the
   EmrStepOperator and EmrStepSensor


Possible approaches

   - This could be achieved by offloading actual logic to Hooks and then
   using as many hooks as needed within an operator
   - A hacky alternative (if at all) would be SubDagOperator



   - Are hooks the right tool for this problem?
   - Any other way to compose operators together?
   - Is it a good idea to combine operators at all?

Here's <> my complete (more
elaborate) question on StackOverflow


*Shubham Gupta*
Software Engineer

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message