From dev-return-4783-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Fri Mar 30 08:07:18 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D619C180647 for ; Fri, 30 Mar 2018 08:07:17 +0200 (CEST) Received: (qmail 90782 invoked by uid 500); 30 Mar 2018 06:07:16 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 90762 invoked by uid 99); 30 Mar 2018 06:07:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Mar 2018 06:07:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 0F3AF1A17A9 for ; Fri, 30 Mar 2018 06:07:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id aeWGV6kp5UiG for ; Fri, 30 Mar 2018 06:07:14 +0000 (UTC) Received: from mail-io0-f196.google.com (mail-io0-f196.google.com [209.85.223.196]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id BDFF25F175 for ; Fri, 30 Mar 2018 06:07:13 +0000 (UTC) Received: by mail-io0-f196.google.com with SMTP id d6so8164671iog.1 for ; Thu, 29 Mar 2018 23:07:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=W/KG8M1YysWQK9D8KRPaJ+LsWt8+/9aLH9scpzCK+zI=; b=hkGrvv2OHXxFXC+oTuYOgYX5e01/q1e8MZ0OdCixt54viReC1lTpTYlspSXB8PgR6d OzxY043t8adMFdxs9dHtuD43niWuMwATSJS2219K2mPSiIiyh5N/69AyWFDjx1Ndqio9 RZzhfEkg4FSppxR6kYiVPhhzqZF904JExyLvNuEVczy69WDh8jHjGdOEB1hUSNdd3BgF njXEhMvIE3n5ycNLZDoYbhHDGGC7C6UseNoYkPR+1EaeZmMXo/6IwduLfkSFb4s/0P6i FnbJ0ceOrFbW2ohXTOrdC6yuM9FJCbZd9jeI/ysdzWgEReUJVh5Q8P3ZZD/m5cwvWtya K5Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=W/KG8M1YysWQK9D8KRPaJ+LsWt8+/9aLH9scpzCK+zI=; b=XwMoWw90L3n44NaFKJB6qNVwy+C3/a4JD1JNKHd2etYNGMJ3h13muG3YueFEREImcv 8hyAAtz6MeTSNV9qOhw1LlrAFDLLYR9Y63DvpGCxFIGjwiKKSruRpZeCr55wtciDx9wf DMjIJmjwlwCMV44F+DKDpIAK/SL8eZiRh+ym/6RNZXkQPxukJukei1SYfFIkCcmmEJcP j72yGMm/at6bWgQwD1HgXhskRV6l0EItwgtSdsCkJsYUvIZZaUbDohLCIT///i5mWyR7 NbMgUux3/7r6afFI7J8nH6fO05slgQ77LgI68VIvao8YExoArnjSPUoQbhj0nGp681EN MKPQ== X-Gm-Message-State: AElRT7HgjtjsiN1FaR2OPPreW7Z0s5t5MWUizrPLoyPHjUoIwLJa/eC/ k7sNrAMt9pqDAmyoN6Dba69GHv3VOSVciby+vw4Jrw== X-Google-Smtp-Source: AG47ELvSfBHP94OtJ8RjkunF6TaKdaw2VDKxw+RLwwLpM4KBsZBvWDcs/7gwkCj7R3k1n4WM1l/MB2+cc9MjucESAzs= X-Received: by 10.107.14.136 with SMTP id 130mr43325396ioo.170.1522390032897; Thu, 29 Mar 2018 23:07:12 -0700 (PDT) MIME-Version: 1.0 References: <7A334ABD-A3F8-4E5B-A603-98B8FFF7E1B3@gmail.com> In-Reply-To: From: Alex Tronchin-James 949-412-7220 Date: Fri, 30 Mar 2018 06:07:01 +0000 Message-ID: Subject: Re: What are the advantages of plugins, not sure I see any? To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="001a113985da1edc0505689b0d28" --001a113985da1edc0505689b0d28 Content-Type: text/plain; charset="UTF-8" At Netflix we've put our plugin inside the DAGs folder and pointed the config to it there so we can both import directly in DAGs AND update the plugin as we go. This makes it easy to test changes to operators needed for ongoing DAG development in the same PR. The two plugin features I've used which don't translate to the direct import approach are custom macros (we provide some internal libraries) and UI menu links, which we use for linking local docs describing our deployment and custom operators, server/worker monitoring with atlas, and genie job monitoring. On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman wrote: > It's presumably useful if you want to package your plugins for other people > to use but it seems like everyone just adds those directly to the Airflow > codebase these days. > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin wrote: > > > Yeah so far I have only written hooks and operators so maybe the benefit > > only kicks in for other airflow abstractions. > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman < > > george@cloverhealth.com.INVALID> wrote: > > > > > > We also import our operators and sensors directly. > > > > > > However, executors and some other pieces are a little bit harder to > deal > > > with as non-plugins > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin > > wrote: > > >> > > >> Hello, > > >> > > >> I just got done writing a few plugins, and the process has left me > > >> wondering what the real benefits are? As far as I can tell, it makes > > >> testing more difficult since you cannot import from the created > module, > > you > > >> have to import directly from the plugin. Additionally, your code > editor > > >> isn't aware of these new plugin modules since they are created when > you > > >> start the app up, this makes it seem like there are errors when there > > >> aren't. Why not just create a lib/ dir with hooks, operators etc.. > dirs > > >> inside and be done with it? Very curious what peoples thoughts are, > who > > >> knows I could be testing wrong or writing the plugins wrong. Thanks in > > >> advance! > > >> > > > --001a113985da1edc0505689b0d28--