From dev-return-6526-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Fri Sep 14 19:20:10 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4A95E180647 for ; Fri, 14 Sep 2018 19:20:09 +0200 (CEST) Received: (qmail 66415 invoked by uid 500); 14 Sep 2018 17:20:08 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 66393 invoked by uid 99); 14 Sep 2018 17:20:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2018 17:20:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 1B2B3C0331 for ; Fri, 14 Sep 2018 17:20:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.112 X-Spam-Level: X-Spam-Status: No, score=-0.112 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_HIGH=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=opentable.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id ELLs-LUH3nl6 for ; Fri, 14 Sep 2018 17:20:03 +0000 (UTC) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0104.outbound.protection.outlook.com [104.47.42.104]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5C75D5F3EE for ; Fri, 14 Sep 2018 17:20:02 +0000 (UTC) Received: from DM6PR07MB5259.namprd07.prod.outlook.com (20.176.113.205) by DM6PR07MB5641.namprd07.prod.outlook.com (20.178.2.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1101.17; Fri, 14 Sep 2018 17:19:54 +0000 Received: from DM6PR07MB5259.namprd07.prod.outlook.com ([fe80::29b6:4f1b:9043:5ff7]) by DM6PR07MB5259.namprd07.prod.outlook.com ([fe80::29b6:4f1b:9043:5ff7%2]) with mapi id 15.20.1080.021; Fri, 14 Sep 2018 17:19:54 +0000 From: Frank Maritato To: "dev@airflow.incubator.apache.org" Subject: Re: [External] Re: Dynamic tasks in a dag? Thread-Topic: [External] Re: Dynamic tasks in a dag? Thread-Index: AQHUTES13aCMgOoQxE6NPlheTP566KTv8iAA//+OjACAAH7YAP//kOaA Date: Fri, 14 Sep 2018 17:19:54 +0000 Message-ID: <734A33DD-10DB-4376-AC12-A4AEBD289309@opentable.com> References: <6309CD01-7C3A-4EAE-AE57-9742F4CC542C@opentable.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [107.185.109.155] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM6PR07MB5641;6:OSkwfiB0++mJdLO4S7p1BYI55ThlA7LaYt2umHBnkoW3oK4N1KtSeI+gRrpQxSuFZCJVKykxFEoL/8YpYRIX7CJ/kS4RH1fclyZldbrsYmMKU4mqs9NkNeuddf8JfezStPwTBsraGLVBh6TFItZpl8/3axPkEReO+EBlZklcn/4o9YqEqdQW/gycHodMnpYWUaYXjd41fFNktXU51MZd5A2csL3RVgKHaURubyDBApdwhg5KK6i72bdM3g650Erpvev2xOd1n6b6ue3837McPdFBOf2Oe6EvGReozVsI9ymQMzMTyTZJdFHhiKCwLdiMYZOtTSCF86vp8SLPXHycjaGXNMGvNS4YSb3+LQDpwXaPKcyNnpWs7EHJFOpr1Bzaeje2Q89iCvKlgbsF8LCRSNK6LDNcIUfiOTwwb0rWJe6u96la3KRB7TbQ1EypjhAZ3n2WPeKCvnBYJxNzLwY5Vw==;5:ulOH3wgFfx2o94uEY3CNtwAAZ8Vmd+JKzjTHSdgXD8hxR5e0dJzkyW90nX/eDbi6M6nPUD56GNwQ+BeYiMRKGXMORvon2jG8aLsAZi92bRCKFexUgbxRi6YEDsh0Pglkj00YT0Fgwpk4/pNy+cAGyRa8E3Xuxr1MVZgSS4IHPPE=;7:HWHrPtCeesE8PYLT3/M8ykYONx/tS0odHyGLv3towpZHIVAk4PPRW5x0605rmaTRieLeCNrA/0dLYOOpQasw1LwTENv8uvi8VKBtCYW0p0l1Cjz1cztwENWHyaHpUz7KUr9v0YmUkBaCZLlZUNS6W1/o3AlVQ1FKE49kI4Q+eVWTE7aNqqPUQ2xgWbnaZFcvZ22ea8i3QgzpBhcLLNNxTMnsh+OjbpfT0Eb7IZgDF7QZ8bwE+HlCpCEKWneVgUFS x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: a4d7c76a-1d16-4424-0842-08d61a6649cb x-microsoft-antispam: BCL:0;PCL:0;RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(2017052603328)(7153060)(7193020);SRVR:DM6PR07MB5641; x-ms-traffictypediagnostic: DM6PR07MB5641: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(271806183753584)(85827821059158)(265273979862326)(788757137089)(93667724000731); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231344)(944501410)(52105095)(3002001)(149027)(150027)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201708071742011)(7699050)(76991041);SRVR:DM6PR07MB5641;BCL:0;PCL:0;RULEID:;SRVR:DM6PR07MB5641; x-forefront-prvs: 07954CC105 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39860400002)(366004)(376002)(136003)(396003)(346002)(189003)(199004)(51874003)(6506007)(446003)(3846002)(5660300001)(66066001)(6346003)(6436002)(102836004)(26005)(305945005)(25786009)(83716003)(82746002)(33656002)(6486002)(2501003)(76176011)(2906002)(53546011)(6116002)(6246003)(8936002)(68736007)(5250100002)(86362001)(53936002)(81166006)(186003)(81156014)(5640700003)(1730700003)(7736002)(99286004)(2616005)(476003)(486006)(105586002)(8676002)(2351001)(2900100001)(6512007)(256004)(478600001)(106356001)(11346002)(93886005)(14454004)(6916009)(316002)(229853002)(36756003)(97736004);DIR:OUT;SFP:1102;SCL:1;SRVR:DM6PR07MB5641;H:DM6PR07MB5259.namprd07.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: opentable.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=fmaritato@opentable.com; x-microsoft-antispam-message-info: m8PA9opVnt8uXPJzJE4CDa9BNePEDBj1yat1Uf7tnXOtD6AzREvGG6q16ZzSAZjOMUfUSjGvlgnhghQ9NMuA5XHE3K5zKF/uyU/NiKmjO+Vz8NE+BJ5esdMwoBdjHN2bocRzcN4xj1CbilzHkIcRv+CfAmyROhPCXJVtrET2BBOrWtiegVquUZJjgkjDAFR3d+AwhySkxvr5uMbDLM5oKE6g8g15XUCA0RPKEdf8cjz5TZBhJRy8T+D4tPCWx6W2txXFJlmeLdMtI5CbMRfdiUouIWwUzSCODaCvTM7hE9uwNlJXKOrG2eqwADFsrhxTPMAaM5naxjzrB8D0YK8NWQ7goD0x/cSWdVLWWymbwiI= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <2053AF3E0BBB65459B61C9F402DED3C1@namprd07.prod.outlook.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-OriginatorOrg: opentable.com X-MS-Exchange-CrossTenant-Network-Message-Id: a4d7c76a-1d16-4424-0842-08d61a6649cb X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Sep 2018 17:19:54.4970 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: fd157fd5-1236-4610-b786-2578dee6ff2c X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR07MB5641 T2ssIG15IG1pc3Rha2UuIEkgdGhvdWdodCB0aGF0IGNvbW1hbmQgd2FzIHF1ZXJ5aW5nIHRoZSBz ZXJ2ZXIgZm9yIGl0cyBpbmZvcm1hdGlvbiBhbmQgbm90IGp1c3QgbG9va2luZyBpbiBhIGRpcmVj dG9yeSByZWxhdGl2ZSB0byB3aGVyZSBpdCBpcyBiZWluZyBydW4uIEkgaGF2ZSBpdCB3b3JraW5n IG5vdy4gVGhhbmtzIENocmlzIGFuZCBTYWkhDQoNCg0K77u/T24gOS8xNC8xOCwgOTo1OCBBTSwg IkNocmlzIFBhbG1lciIgPGNocmlzQGNycGFsbWVyLmNvbT4gd3JvdGU6DQoNCiAgICBUaGUgcmVs YXRpdmUgcGF0aHMgbWlnaHQgd29yayBmcm9tIHdoZXJlIGV2ZXIgeW91IGFyZSBldm9raW5nICdh aXJmbG93DQogICAgbGlzdF90YXNrcycsIGJ1dCB0aGF0IGRvZXNuJ3QgbWVhbiB0aGV5IHdvcmsg ZnJvbSB3aGVyZXZlciB0aGUgd2Vic2VydmVyIGlzDQogICAgcGFyc2luZyB0aGUgZGFncyBmcm9t Lg0KICAgIA0KICAgIERvZXMgcnVubmluZyAnYWlyZmxvdyBsaXN0X3Rhc2tzJyBmcm9tIHNvbWUg b3RoZXIgcnVubmluZyBkaXJlY3Rvcnkgd29yaz8NCiAgICANCiAgICBPbiBGcmksIFNlcCAxNCwg MjAxOCBhdCAxMjozNSBQTSBGcmFuayBNYXJpdGF0bw0KICAgIDxmbWFyaXRhdG9Ab3BlbnRhYmxl LmNvbS5pbnZhbGlkPiB3cm90ZToNCiAgICANCiAgICA+IERvIHlvdSBtZWFuIGdpdmUgdGhlIGZ1 bGwgcGF0aCB0byB0aGUgZmlsZXM/IFRoZSByZWxhdGl2ZSBwYXRoIEknbSB1c2luZw0KICAgID4g ZGVmaW5pdGVseSB3b3Jrcy4gV2hlbiBJIHR5cGUgYWlyZmxvdyBsaXN0X2RhZ3MsIEkgY2FuIHNl ZSB0aGUgb3V0cHV0IGZyb20NCiAgICA+IHRoZSBwcmludCBzdGF0ZW1lbnRzIHRoYXQgdGhlIGds b2IgaXMgZmluZGluZyBteSBzcWwgZmlsZXMgYW5kIGNyZWF0aW5nIHRoZQ0KICAgID4gc25vd2Zs YWtlIG9wZXJhdG9ycy4NCiAgICA+DQogICAgPiBhaXJmbG93IGxpc3RfdGFza3Mgd29ya2Zsb3cg YWxzbyBsaXN0cyBhbGwgdGhlIG9wZXJhdG9ycyBJJ20gY3JlYXRpbmcuIEknbQ0KICAgID4ganVz dCBub3Qgc2VlaW5nIHRoZW0gaW4gdGhlIHVpLg0KICAgID4NCiAgICA+IE9uIDkvMTQvMTgsIDk6 MTAgQU0sICJTYWkgUGhhbmluZGhyYSIgPHBoYW5pODk5NkBnbWFpbC5jb20+IHdyb3RlOg0KICAg ID4NCiAgICA+ICAgICBIaSBmcmFuaywNCiAgICA+ICAgICBDYW4geW91IHRyeSBnaXZpbmcgZ2xv YmFsIHBhdGhzPw0KICAgID4NCiAgICA+ICAgICBPbiBGcmkgMTQgU2VwLCAyMDE4LCAyMTozNSBG cmFuayBNYXJpdGF0bywgPGZtYXJpdGF0b0BvcGVudGFibGUuY29tDQogICAgPiAuaW52YWxpZD4N CiAgICA+ICAgICB3cm90ZToNCiAgICA+DQogICAgPiAgICAgPiBIaSwNCiAgICA+ICAgICA+DQog ICAgPiAgICAgPiBJJ20gdXNpbmcgYXBhY2hlIGFpcmZsb3cgMS4xMC4wIGFuZCBJJ20gdHJ5aW5n IHRvIGR5bmFtaWNhbGx5DQogICAgPiBnZW5lcmF0ZQ0KICAgID4gICAgID4gc29tZSB0YXNrcyBp biBteSBkYWcgYmFzZWQgb24gZmlsZXMgdGhhdCBhcmUgaW4gdGhlIGRhZ3MgZGlyZWN0b3J5Lg0K ICAgID4gVGhlDQogICAgPiAgICAgPiBwcm9ibGVtIGlzLCBJIGRvbid0IHNlZSB0aGVzZSB0YXNr cyBpbiB0aGUgdWksIEkganVzdCBzZWUgdGhlDQogICAgPiAnc3RhcnQnIGR1bW15DQogICAgPiAg ICAgPiBvcGVyYXRvci4gSWYgSSB0eXBlICdhaXJmbG93IGxpc3RfdGFza3Mgd29ya2Zsb3cnLCB0 aGV5IGFyZSBsaXN0ZWQuDQogICAgPiAgICAgPiBUaG91Z2h0cz8NCiAgICA+ICAgICA+DQogICAg PiAgICAgPiBIZXJlIGlzIGhvdyBJJ20gZ2VuZXJhdGluZyB0aGUgdGFza3M6DQogICAgPiAgICAg Pg0KICAgID4gICAgID4NCiAgICA+ICAgICA+IGRlZiBjcmVhdGVfc25vd2ZsYWtlX29wZXJhdG9y KGZpbGUsIGRhZywgc25vd2ZsYWtlX2Nvbm5lY3Rpb24pOg0KICAgID4gICAgID4gICAgIGZpbGVf cmVwbCA9IGZpbGUucmVwbGFjZSgnLycsICdfJykNCiAgICA+ICAgICA+ICAgICBmaWxlX3JlcGwg PSBmaWxlX3JlcGwucmVwbGFjZSgnLnNxbCcsICcnKQ0KICAgID4gICAgID4gICAgIHByaW50KCJU QVNLX0lEIHt9Ii5mb3JtYXQoZmlsZV9yZXBsKSkNCiAgICA+ICAgICA+ICAgICByZXR1cm4gU25v d2ZsYWtlT3BlcmF0b3IoDQogICAgPiAgICAgPiAgICAgICAgIGRhZz1kYWcsDQogICAgPiAgICAg PiAgICAgICAgIHRhc2tfaWQ9J2NyZWF0ZV97fScuZm9ybWF0KGZpbGVfcmVwbCksDQogICAgPiAg ICAgPiAgICAgICAgIHNub3dmbGFrZV9jb25uX2lkPXNub3dmbGFrZV9jb25uZWN0aW9uLA0KICAg ID4gICAgID4gICAgICAgICBzcWw9ZmlsZQ0KICAgID4gICAgID4gICAgICkNCiAgICA+ICAgICA+ DQogICAgPiAgICAgPiBEQUdfTkFNRSA9ICdjcmVhdGVfb2JqZWN0cycNCiAgICA+ICAgICA+IGRh ZyA9IERBRygNCiAgICA+ICAgICA+ICAgICBEQUdfTkFNRSwNCiAgICA+ICAgICA+ICAgICBkZWZh dWx0X2FyZ3M9YXJncywNCiAgICA+ICAgICA+ICAgICBkYWdydW5fdGltZW91dD10aW1lZGVsdGEo aG91cnM9MiksDQogICAgPiAgICAgPiAgICAgc2NoZWR1bGVfaW50ZXJ2YWw9Tm9uZSwNCiAgICA+ ICAgICA+ICkNCiAgICA+ICAgICA+DQogICAgPiAgICAgPiBzdGFydCA9IER1bW15T3BlcmF0b3Io DQogICAgPiAgICAgPiAgICAgZGFnPWRhZywNCiAgICA+ICAgICA+ICAgICB0YXNrX2lkPSJzdGFy dCIsDQogICAgPiAgICAgPiApDQogICAgPiAgICAgPg0KICAgID4gICAgID4gcHJpbnQoImNyZWF0 aW5nIHNub3dmbGFrZSBvcGVyYXRvcnMiKQ0KICAgID4gICAgID4NCiAgICA+ICAgICA+IGZvciBm aWxlIGluIGdsb2IoJ2RhZ3Mvc25vd3NxbC9jcmVhdGUvdWRmLyouc3FsJyk6DQogICAgPiAgICAg PiAgICAgcHJpbnQoIkZJTEUge30iLmZvcm1hdChmaWxlKSkNCiAgICA+ICAgICA+ICAgICB0YXNr ID0gY3JlYXRlX3Nub3dmbGFrZV9vcGVyYXRvcihmaWxlLCBkYWcsICdzbm93Zmxha2VfZGVmYXVs dCcpDQogICAgPiAgICAgPiAgICAgdGFzay5zZXRfdXBzdHJlYW0oc3RhcnQpDQogICAgPiAgICAg Pg0KICAgID4gICAgID4gZm9yIGZpbGUgaW4gZ2xvYignZGFncy9zbm93c3FsL2NyZWF0ZS90YWJs ZS8qLnNxbCcpOg0KICAgID4gICAgID4gICAgIHByaW50KCJGSUxFIHt9Ii5mb3JtYXQoZmlsZSkp DQogICAgPiAgICAgPiAgICAgdGFzayA9IGNyZWF0ZV9zbm93Zmxha2Vfb3BlcmF0b3IoZmlsZSwg ZGFnLCAnc25vd2ZsYWtlX2RlZmF1bHQnKQ0KICAgID4gICAgID4gICAgIHRhc2suc2V0X3Vwc3Ry ZWFtKHN0YXJ0KQ0KICAgID4gICAgID4NCiAgICA+ICAgICA+IGZvciBmaWxlIGluIGdsb2IoJ2Rh Z3Mvc25vd3NxbC9jcmVhdGUvdmlldy8qLnNxbCcpOg0KICAgID4gICAgID4gICAgIHByaW50KCJG SUxFIHt9Ii5mb3JtYXQoZmlsZSkpDQogICAgPiAgICAgPiAgICAgdGFzayA9IGNyZWF0ZV9zbm93 Zmxha2Vfb3BlcmF0b3IoZmlsZSwgZGFnLCAnc25vd2ZsYWtlX2RlZmF1bHQnKQ0KICAgID4gICAg ID4gICAgIHRhc2suc2V0X3Vwc3RyZWFtKHN0YXJ0KQ0KICAgID4gICAgID4NCiAgICA+ICAgICA+ IHByaW50KCJkb25lIHt9Ii5mb3JtYXQoc3RhcnQuZG93bnN0cmVhbV90YXNrX2lkcykpDQogICAg PiAgICAgPg0KICAgID4gICAgID4gVGhhbmtzIGluIGFkdmFuY2UNCiAgICA+ICAgICA+IC0tDQog ICAgPiAgICAgPiBGcmFuayBNYXJpdGF0bw0KICAgID4gICAgID4NCiAgICA+DQogICAgPg0KICAg ID4NCiAgICANCg0K