airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Gupta <y2k.shubhamgu...@gmail.com>
Subject Fusing operators together
Date Wed, 14 Nov 2018 20:56:07 GMT
*[Please let me know if this is NOT the correct place for such a query]*

Hello maintainers and committers,
I've stumbled upon this design decision for my Airflow project. Any
pointers would be helpful.

Overview

   - I'm in the process of deploying Airflow and I've felt the need to
   merge groups of operators that form a single logical task (to clear the
   clutter in huge DAGs)
   - The most common use-case would be coupling an operator and the
   corresponding sensor. For instance, one might want to chain together the
   EmrStepOperator and EmrStepSensor


----

Possible approaches

   - This could be achieved by offloading actual logic to Hooks and then
   using as many hooks as needed within an operator
   - A hacky alternative (if at all) would be SubDagOperator


----

Questions

   - Are hooks the right tool for this problem?
   - Any other way to compose operators together?
   - Is it a good idea to combine operators at all?


Here's <https://stackoverflow.com/questions/53308306> my complete (more
elaborate) question on StackOverflow

Thanks

*Shubham Gupta*
Software Engineer
 zomato

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message