From dev-return-7000-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Wed Nov 14 21:56:50 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4241918066B for ; Wed, 14 Nov 2018 21:56:50 +0100 (CET) Received: (qmail 93354 invoked by uid 500); 14 Nov 2018 20:56:49 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 93337 invoked by uid 99); 14 Nov 2018 20:56:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Nov 2018 20:56:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 186DBD572D for ; Wed, 14 Nov 2018 20:56:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.35 X-Spam-Level: X-Spam-Status: No, score=0.35 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-1.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_FONT_LOW_CONTRAST=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id hgGosGdCAdDG for ; Wed, 14 Nov 2018 20:56:46 +0000 (UTC) Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id EAEC35FBA8 for ; Wed, 14 Nov 2018 20:56:45 +0000 (UTC) Received: by mail-ot1-f45.google.com with SMTP id u16so15753850otk.8 for ; Wed, 14 Nov 2018 12:56:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=2+amP1JABGpktCWvDI5eUi4wsltfI7rSVyWcAmy8pNI=; b=UsXYMfeW6Frr9m6/oe3xCnQC9G3qw9fy2xX4dR4wVLIhsZ3+V7B/r/+2cGQeUO8t+h 9ilkzSrI/ozTGfabQv9rE5Pxvh/rMOLHkq6qm8JeVt8+m981rI1q/Cq+aZmIU1oEXbZq PYzcU1DIcKzUfgGz/fUmRl/DVjrR+iEwjSMoTf3mdR2wEvZiAPhubqY18HkXIRjrZicy qe2rp0nkXx/JibW5KdJ93mJn16zuremaK/+ZvWQeJMrpMcxkarAz+NGYNCzrs5UVfYl6 AFcj3LteqfjJ5hMixpXzBVk4RKGoWsdERLd3Km/ufFU+XpCwcYdj5crZ6FNjqemBztyt CmGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=2+amP1JABGpktCWvDI5eUi4wsltfI7rSVyWcAmy8pNI=; b=DpzXNPGogF5Kne8m+Eaz3vAEWInTmvjN9js/ClS6dpG3V5ntZJcqxUb3ZOiIDIkhC7 xfTAlcfLxD93emJ431oVRo3LoqL1LWR4R0dZ0bCD1PNF4QF2f3Uqj/Czc2GLjOf1zoAF Uuff/hNz1Sd1XbsWnkrX78AFz8xrmgfqdh5hap37UV+P1SGJ4/j708b3jCIXDU4eUFR8 Ukv4fPVXN/5gMVt263O+fM7iVLRWwlN0dRptxY+Ww659PfJU2FvSlI78EYxHXewzC5Bm J/jfC+Vj1Hb7TsEqlcr6nSaLibC5HLYfWzas/+R9qj1HIg78cinVNXz8G7BGzUOZpS+q X+EA== X-Gm-Message-State: AGRZ1gLVWdxX7wncEjqdfYzQOuv3wqZGYZmrz7KiUbCYnynV09rVWJNn KfLdSxtZvucEkiiYd6j58v7KGw7E+FnXs3BsZ+35mQ== X-Google-Smtp-Source: AJdET5fm+xDiTL6ttn300kzqRawZfDF8eDLv9h+874PMqdAwfp8Os0a+CdoS+5b0iRlShG44sF0uFaYP3ZQl5gGFsYs= X-Received: by 2002:a9d:156d:: with SMTP id z42mr2108248otz.201.1542229004286; Wed, 14 Nov 2018 12:56:44 -0800 (PST) MIME-Version: 1.0 From: Shubham Gupta Date: Thu, 15 Nov 2018 02:26:07 +0530 Message-ID: Subject: Fusing operators together To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="000000000000f680c4057aa62bcf" --000000000000f680c4057aa62bcf Content-Type: text/plain; charset="UTF-8" *[Please let me know if this is NOT the correct place for such a query]* Hello maintainers and committers, I've stumbled upon this design decision for my Airflow project. Any pointers would be helpful. Overview - I'm in the process of deploying Airflow and I've felt the need to merge groups of operators that form a single logical task (to clear the clutter in huge DAGs) - The most common use-case would be coupling an operator and the corresponding sensor. For instance, one might want to chain together the EmrStepOperator and EmrStepSensor ---- Possible approaches - This could be achieved by offloading actual logic to Hooks and then using as many hooks as needed within an operator - A hacky alternative (if at all) would be SubDagOperator ---- Questions - Are hooks the right tool for this problem? - Any other way to compose operators together? - Is it a good idea to combine operators at all? Here's my complete (more elaborate) question on StackOverflow Thanks *Shubham Gupta* Software Engineer zomato --000000000000f680c4057aa62bcf--