From dev-return-6584-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Tue Sep 18 19:28:52 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F08EB180672 for ; Tue, 18 Sep 2018 19:28:51 +0200 (CEST) Received: (qmail 68508 invoked by uid 500); 18 Sep 2018 17:28:50 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 68483 invoked by uid 99); 18 Sep 2018 17:28:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Sep 2018 17:28:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A971DC206A for ; Tue, 18 Sep 2018 17:28:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.991 X-Spam-Level: * X-Spam-Status: No, score=1.991 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=driesprong-frl.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5ZAP8uuwEjGM for ; Tue, 18 Sep 2018 17:28:46 +0000 (UTC) Received: from mail-yb1-f193.google.com (mail-yb1-f193.google.com [209.85.219.193]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 52A845F1C6 for ; Tue, 18 Sep 2018 17:28:45 +0000 (UTC) Received: by mail-yb1-f193.google.com with SMTP id e190-v6so1171380ybb.5 for ; Tue, 18 Sep 2018 10:28:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=driesprong-frl.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=t3TVFSmSJqPZyzugecbAhO51jny0EjItNyZV4hhMTik=; b=nzWimt53BnFj3HGhw6MGWzP5mmOelwRwjw+SOgcpWXovLLb/xCcCrHU/Z1jmqNFu6k SLEUZcQytIa9zl8PRKNjVmQZJf/6gAAi5jRZUoxG5H2yqErY6dv4KjefApdcbmY14tsG sVGdJtklRsWykvidNSWjZ/xmQlSI8ABzBGEThY1i2w5N4dU9W3jfRCo0HSnXIDccKjOl 9fuC/6sFw09bDVffXuB4SAC1YD0XmSwKBoHSYCyOvaNbeHC525e3JDKaJBzCrekM5KdS kJ9EeJOThYaTNQ7+nM8eeHr9AuyG4OcgrVhH6K3yqdJOzQxNuTbOvNgUAIgWR7CplTcC GJbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=t3TVFSmSJqPZyzugecbAhO51jny0EjItNyZV4hhMTik=; b=fHcMA5xVRp/yxOOl6wfRcTMi60f7LdANukTLXTOb6mOu7CTvrYP3LW0RcM8tijW4F0 Pib7EPrJSkIVqWuSIHQcss/qxCByHk/a6jkDQW42+zYKRM9DQfiag9L2DseAIIeHIwZ3 dWudQmho+NsRvx9c/yUMhyBIV1WixZja1hF7lXuV+lxNlkGU3Es040+OrBtV67IGJUBj lwaUqs1shezyhfvs/WxdkZbB3AiYuBIM4NIEsTG14UhvtL+afqHIseKkuE/H0DgD+IlY V4FNlIMZLEITRdCi+He5hN/dO/W1g3IQHnjoDJ7GtGbLsVdHE47xjR2AWYj2ZMeg4eeP 2Tzw== X-Gm-Message-State: APzg51AAlNG5hfLqhgEurZ+u/dLWAPQFQP2fss+79iX9biqxoD4SVy0e cb0Proht28TuDr7xtu/c6R34GbFT1uxN/NvX9UzDGks4zW0= X-Google-Smtp-Source: ANB0VdaY4UfRkyoEFHIthwULZ3vqomj4PIuMV727zyOCCQ/hjXCAjrSWZt4TFdNpcT9crbNY3Y17Rz7cayZBSpmLUpo= X-Received: by 2002:a25:ba4b:: with SMTP id z11-v6mr13229218ybj.427.1537291723152; Tue, 18 Sep 2018 10:28:43 -0700 (PDT) MIME-Version: 1.0 References: <308670DB-BD2A-4738-81B1-3F6FB312C0C8@apache.org> <5E8FA0D4-2056-4377-B3F4-98412FD51B4F@amplify-analytics.com> In-Reply-To: <5E8FA0D4-2056-4377-B3F4-98412FD51B4F@amplify-analytics.com> From: "Driesprong, Fokko" Date: Tue, 18 Sep 2018 19:28:31 +0200 Message-ID: Subject: Re: Guidelines on Contrib vs Non-contrib To: dev@airflow.incubator.apache.org Content-Type: multipart/alternative; boundary="0000000000001337b40576289fef" --0000000000001337b40576289fef Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I fully agree with using plain Python modules :) I don't think a lot of hooks/operators graduate to core since it will break the import. A few of them, for example Databricks and the Google hooks are mature enough. For me the main point is having test coverage and a stable API. Cheers, Fokko Op di 18 sep. 2018 om 18:30 schreef Victor Noagbodji < vnoagbodji@amplify-analytics.com>: > yes, please! > > > On Sep 18, 2018, at 12:23 PM, Maxime Beauchemin < > maximebeauchemin@gmail.com> wrote: > > > > +1 for deprecating operators/hooks as plugins, let's use Python's good > old > > python packages and maybe python "entry points" if we want to inject th= em > > in "airflow.operators"/"airflow.hooks" (which is probably not necessary= ) > > > > On Tue, Sep 18, 2018 at 2:12 AM Ash Berlin-Taylor > wrote: > > > >> Operators and hooks don't need any special plugin system - simply havi= ng > >> them as as separate Python modules which are imported using normal > python > >> semantics is enough. > >> > >> In fact now that I think about it: I want to deprecate the plugins > >> registering hooks/operators etc and limit it to only bits which a simp= le > >> python import can't manage - which I think is only anything that needs > to > >> be registered with another system, such as custom routes in the web UI= . > >> > >> I'll draft an AIP for this soon. > >> > >> -ash > >> > >> > >>> On 18 Sep 2018, at 00:50, George Leslie-Waksman > >> wrote: > >>> > >>> Given we have a plugin system, could we alternatively move away from > >>> keeping non-core supported code outside of the core project/repo? > >>> > >>> It would hugely decrease the surface area of the main repository and > >>> testing infrastructure to get most of the contrib code out to its own > >> place. > >>> > >>> Further, it would decrease the committer burden of having to > >> approve/merge > >>> code that is not supposed to be their responsibility. > >>> > >>> On Mon, Sep 17, 2018 at 4:37 PM Tim Swast > >> wrote: > >>> > >>>>> Individual operators and hooks living in separate repositories on > >> github > >>>> (or possibly other Apache projects), which are then distributed by p= ip > >> and > >>>> installed as libraries seems like it would scale better. > >>>> > >>>> Pandas did this about a year ago, and it's seemed to have worked wel= l. > >> For > >>>> example, pandas.read_gbq is a very thin wrapper around > >> pandas_gbq.read_gbq > >>>> (distributed as a separate package). It has made it easier for me to > >> track > >>>> issues corresponding to my area of expertise. > >>>> > >>>> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan > wrote: > >>>> > >>>>>> My understanding as a contributor is that if a hook/operator is in > >>>> core, > >>>>> it > >>>>>> means that a committer is willing to take personal responsibility = to > >>>>>> maintain it (or at least help maintain it), and everything else go= es > >> in > >>>>>> contrib. > >>>>> > >>>>> That's not correct. All of the code is owned by the entire > >>>>> community[1]; no one person is responsible for any of it. There's = no > >>>>> silos, fiefdoms, walled gardens, etc. If the community cannot > support > >>>>> a piece of code it should be deprecated and subsequently removed. > >>>>> > >>>>> Contrib sections are almost always problematic for this reason. > >>>>> Hadoop ended up abandoning its. Because Airflow acts as a gatherin= g > >>>>> point for so many disparate technologies (databases, storage system= s, > >>>>> compute engines, etc.), trying to keep all of them corralled and up > to > >>>>> date will be very difficult. Individual operators and hooks living > in > >>>>> separate repositories on github (or possibly other Apache projects)= , > >>>>> which are then distributed by pip and installed as libraries seems > >>>>> like it would scale better. > >>>>> > >>>>> -Jakob > >>>>> > >>>>> [1] > >> https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie > >>>>> > >>>>> On 15 September 2018 at 13:29, Jeff Payne > wrote: > >>>>>> How many operators are added to contrib per month? Is it too many = to > >>>>> make the decision case by case? If so, then the above mentioned rul= e > >>>> sounds > >>>>> fairly reasonable. However, if that's the rule, shouldn't a bunch o= f > >>>>> existing modules be moved from contrib to core? > >>>>>> > >>>>>> Get Outlook for Android > >>>>>> > >>>>>> ________________________________ > >>>>>> From: Taylor Edmiston > >>>>>> Sent: Saturday, September 15, 2018 1:13:47 PM > >>>>>> To: dev@airflow.incubator.apache.org > >>>>>> Subject: Re: Guidelines on Contrib vs Non-contrib > >>>>>> > >>>>>> My understanding as a contributor is that if a hook/operator is in > >>>> core, > >>>>> it > >>>>>> means that a committer is willing to take personal responsibility = to > >>>>>> maintain it (or at least help maintain it), and everything else go= es > >> in > >>>>>> contrib. > >>>>>> > >>>>>> *Taylor Edmiston* > >>>>>> Blog | LinkedIn > >>>>>> | Stack Overflow > >>>>>> | > Developer > >>>>> Story > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik > >>>> wrote: > >>>>>> > >>>>>>> Hi, all (mainly contributors), > >>>>>>> > >>>>>>> Can we decide on a common guideline on when a hook/operator shoul= d > go > >>>>> under > >>>>>>> contrib vs core? > >>>>>>> > >>>>>>> Regards, > >>>>>>> > >>>>>>> *Kaxil Naik* > >>>>>>> *Big Data Consultant *@ *Data Reply UK* > >>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark= & > >>>>> Neo4j > >>>>>>> Developer > >>>>>>> *Phone: *+44 (0) 74820 88992 > >>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >>>>>>> > >>>>> > >>>> -- > >>>> * =E2=80=A2 **Tim Swast* > >>>> * =E2=80=A2 *Software Friendliness Engineer > >>>> * =E2=80=A2 *Google Cloud Developer Relations > >>>> * =E2=80=A2 *Seattle, WA, USA > >>>> > >> > >> > > --0000000000001337b40576289fef--