From dev-return-4497-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Tue Jan 30 22:40:39 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 249C618066D for ; Tue, 30 Jan 2018 22:40:39 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 1411A160C53; Tue, 30 Jan 2018 21:40:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 420FE160C2A for ; Tue, 30 Jan 2018 22:40:38 +0100 (CET) Received: (qmail 89955 invoked by uid 500); 30 Jan 2018 21:40:32 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 89943 invoked by uid 99); 30 Jan 2018 21:40:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jan 2018 21:40:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3FE9AD56C3 for ; Tue, 30 Jan 2018 21:40:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=cloverhealth.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Pna3RF9unwl9 for ; Tue, 30 Jan 2018 21:40:29 +0000 (UTC) Received: from mail-qt0-f180.google.com (mail-qt0-f180.google.com [209.85.216.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8378F5F2AD for ; Tue, 30 Jan 2018 21:40:29 +0000 (UTC) Received: by mail-qt0-f180.google.com with SMTP id d8so19265206qtm.0 for ; Tue, 30 Jan 2018 13:40:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=WyhmSZ4EqH8nqvEk07bVp0Wu1ut7T53mqGltvR8n8So=; b=Qxwz9hEU9QczBsQueMM2m46PagHlk1SpFAb8wNCXEdaWy2Xs0Rlw0vxYhxE1atS2qV nJFzO9L7M64APqzrTJkjYZNyYf3Mgjjj9kKmQAMJTleZQS309P2HZgn5IF058LvL7s3E PGlvU6RKW2QadCTOT+pEDuw6FCgK6q+afhChXeI0zwRwu/sym/KsVQb56C2Pk04iwf6i DdOy3G6CbUCkeI7HpEv4x/TcQ2OAuc5YatfUF6hj8/0K7rhjZ/sMFFq4zToTMVgACJYG 42cUc2mSL6ONQpCdl6i8sHUp2TqyNw8ZES6g3NaGyz3Z9u+N+G1u3DQmusJ8CZ80Y2BC HFFA== X-Gm-Message-State: AKwxytcqNwV3P99MphRm20SAMf/p2L4/huZhsPM1VUI47UI6wjLwbChl ELqrvHZXHn7KbjrjH4bqIXIQuNprtITurT1rkykul4Lb X-Google-Smtp-Source: AH8x2276J4FNtVweQc9uZbJvFV1aGxRmD1gVgbENUlochK5UBBKEvOcDo18XvSNrbcWit8MVogNfW/GOD2RDAwSnVGw= X-Received: by 10.200.6.208 with SMTP id j16mr44799789qth.166.1517348428795; Tue, 30 Jan 2018 13:40:28 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: George Leslie-Waksman Date: Tue, 30 Jan 2018 21:40:18 +0000 Message-ID: Subject: Re: Q1 Airflow Bay Area Meetup To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="f403043b2ae419796d05640536ac" --f403043b2ae419796d05640536ac Content-Type: text/plain; charset="UTF-8" If you're still looking for speakers, I have a handful of things I could pull together to talk about: - the interplay between pools, queues, parallelism, dag concurrency, and other job scheduling levers - layering additional instrumentation on top of Airflow operators - integrity testing DAGs in CI before they hit production / unit testing tasks - using Airflow metadata to track systemic execution behavior - how I would have used Airflow if I'd known then what I know now On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire wrote: > +1 on being able to watch a recording, based in Dublin, Ireland but also > very interested in the cloud side of airflow. > > On Thu, 11 Jan 2018, 17:59 Joy Gao, wrote: > > > Great to see that there's a lot of interests! > > > > I will go ahead and work on organizing this event. Details TBD. > > > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz > > wrote: > > > > > I would love to see videos from this as well, as we're east coast. We > > host > > > our airflow installation in GCP using GKE, and would love to compare > > notes. > > > > > > Laura > > > > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin > > wrote: > > > > > > > If it coincides with qcon.ai (april 10-11) or is close to its dates > > > then I > > > > can join. > > > > > > > > B. > > > > > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" >: > > > > > > > > I can give a talk about all the hacks we did to scale Airflow Local > > > > Executor and improve the data pipeline on-call experience at Slack if > > > folks > > > > are interested. > > > > > > > > Regards, > > > > Ananth.P, > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 8 January 2018 at 15:51, George Leslie-Waksman < > > > > george@cloverhealth.com.invalid> wrote: > > > > > > > > > +1 and would love to hear about other folks running Airflow in GCP. > > > > > > > > > > Would strongly prefer March (but I'm just one person) > > > > > > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu > > > > wrote: > > > > > > > > > > > +1 Joy! > > > > > > > > > > > > Would prefer April if possible, I can talk about running Airflow > in > > > GCP > > > > > if > > > > > > there's sufficient interest. > > > > > > > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand > > > wrote: > > > > > > > > > > > > > Sounds great Joy! > > > > > > > > > > > > > > I've promoted you to *Event Organizer* on the Bay Area Apache > > > Airflow > > > > > > > Meetup as per instructions/guidelines on > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups > > > > > > > > > > > > > > Go ahead and start setting up the meetup... we recommend 3 > > speakers > > > > > with > > > > > > > 1-2 of them being external to the hosting company. > > > > > > > > > > > > > > Once the meetup date is setup, we can add it to > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements > > > & > > > > > > tweet > > > > > > > it or share it over the dev list. > > > > > > > -s > > > > > > > > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao > wrote: > > > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > > > I'm Joy from WePay. At the last Airflow meetup in December > > there > > > > was > > > > > a > > > > > > > > demand for hosting a future meetup to cover topics on Airflow > > > > > > > integrations > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS > hooks/operators, > > > best > > > > > > > > practices on running Airflow in the cloud, managed Airflow, > > etc.) > > > > > > > > > > > > > > > > If there is enough interest, WePay can host the next event > > > > sometimes > > > > > in > > > > > > > > March/April. And if you would like to give a talk or have > > another > > > > > topic > > > > > > > > covered, feel free to suggest them as well. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Joy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --f403043b2ae419796d05640536ac--