Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3A92F200D2B for ; Thu, 2 Nov 2017 18:55:23 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 391A0160BE5; Thu, 2 Nov 2017 17:55:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7E75B1609EB for ; Thu, 2 Nov 2017 18:55:22 +0100 (CET) Received: (qmail 98984 invoked by uid 500); 2 Nov 2017 17:55:21 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 98968 invoked by uid 99); 2 Nov 2017 17:55:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Nov 2017 17:55:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 690591A139D for ; Thu, 2 Nov 2017 17:55:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.151 X-Spam-Level: X-Spam-Status: No, score=-0.151 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id pzTxum-hX5PA for ; Thu, 2 Nov 2017 17:55:19 +0000 (UTC) Received: from mail-pf0-f170.google.com (mail-pf0-f170.google.com [209.85.192.170]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 1A2C760D8C for ; Thu, 2 Nov 2017 17:55:19 +0000 (UTC) Received: by mail-pf0-f170.google.com with SMTP id b79so238676pfk.5 for ; Thu, 02 Nov 2017 10:55:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=a6Vqutx2ZZ3sbOVaUOdZQQjxCg3SnPxoAPiGJhmwJ5U=; b=iqjf5JEyZQ4iNPiXDg7dncXvXrnA82rcBU5ALscMF0jwJwiBokaCSJ/Xxx2o7sonYX msVvAKR76nadh9VVfoMC3PrMOWEnY31hGpQYC+IJZJfWJMPd6PIFrneuVmBzKXMRLP1H I4VWNhWxq5SV72iF0bokKb9WcmOEtpQn8KUMsLrImlPoVbTWH/W6nK+j08tU03LQtMug +X7R2j+4Rz0SwckPgUHTzBUdPwFp2kkj3vd3SF69T73GlMEQgMumADbu/2Fk+If6g8jE KaXMvcQD8s88OtEguFYyan459RJC7Tokwx7IV+fHyg/fQBKu+cm68UY4fKQ77x8/c73g 50Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=a6Vqutx2ZZ3sbOVaUOdZQQjxCg3SnPxoAPiGJhmwJ5U=; b=h2ziUA56FI1MmzjXtj0kbRW52Aio6NEsziEKvqbotwa5fqKMjiCMfV0lCF6IYlfbEj nDl6L6iTbxC6V8KNxgoFfLDAk/zgUJZ4e6HlfWU0gzE+SO9rdnWSdQHaqnAYfsiJ2GTT 5aDj3TLyTbSTPQrptKynND+afmtymGLG2iXlzDXg3ezqjg6rQ3D2FkOx6ImKwif3wC04 ZYKx/2Tg8VXpQ9no14NqJ27VrqnS9jUob68FVy+jIjyY902X7Of9wnESsiir/FYWR4v6 udZeBtSAyEFCsQw2KPktcyBnIaLs29dmF58/rpIf8/fZJYTV8hUGrIBlszxGMgRV/jTo fuzQ== X-Gm-Message-State: AMCzsaVDtxKTUqs4MHGoPafkMWxIZJH76fdF7/MHhujL9SWZmveADDHj RQjphyZlD7+KMdJ79c39KvHcL882AXiLgfdnTG3j+A== X-Google-Smtp-Source: ABhQp+Qb/VTgQXzMnrQOyTpiG6TRjq5XIbwcdsLkJivFVJj2sDAV1974SxX9KO+SY1KeDSgqphsir8IQMphCd57QDL0= X-Received: by 10.101.69.197 with SMTP id m5mr4309331pgr.200.1509645311654; Thu, 02 Nov 2017 10:55:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.178.144 with HTTP; Thu, 2 Nov 2017 10:55:11 -0700 (PDT) From: Shoumitra Srivastava Date: Thu, 2 Nov 2017 10:55:11 -0700 Message-ID: Subject: Airflow on ECS To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="089e08221d8c89c17d055d03b0b0" archived-at: Thu, 02 Nov 2017 17:55:23 -0000 --089e08221d8c89c17d055d03b0b0 Content-Type: text/plain; charset="UTF-8" Hi guys, So far we have had a lot of success testing out Airflow and we are now going for a full scale deployment. To that end, we are considering dockerizing airflow and deploying it on one of our ECS clusters. We are planning on separating out the web server and the scheduler to separate tasks and using local executor with an RDS postgres and redis backend. Does anyone else have any suggestions regarding the setup? Any design patterns or good practises and gotchas would be welcome. -Shoumitra --089e08221d8c89c17d055d03b0b0--