From dev-return-6574-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Tue Sep 18 11:12:01 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2D1A4180672 for ; Tue, 18 Sep 2018 11:12:01 +0200 (CEST) Received: (qmail 27932 invoked by uid 500); 18 Sep 2018 09:12:00 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 27921 invoked by uid 99); 18 Sep 2018 09:12:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Sep 2018 09:12:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9D6C1C17A4 for ; Tue, 18 Sep 2018 09:11:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.973 X-Spam-Level: X-Spam-Status: No, score=0.973 tagged_above=-999 required=6.31 tests=[KAM_SHORT=0.001, SPF_SOFTFAIL=0.972] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id a_yulUT5NmDV for ; Tue, 18 Sep 2018 09:11:58 +0000 (UTC) Received: from mail.firemirror.com (mail.firemirror.com [178.63.242.223]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 9634C5F4A2 for ; Tue, 18 Sep 2018 09:11:58 +0000 (UTC) Received: from [217.169.25.231] (helo=[192.168.152.66]) by mail.firemirror.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1g2C2y-00005m-An for dev@airflow.incubator.apache.org; Tue, 18 Sep 2018 09:11:57 +0000 From: Ash Berlin-Taylor Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Guidelines on Contrib vs Non-contrib Date: Tue, 18 Sep 2018 10:11:51 +0100 References: To: dev@airflow.incubator.apache.org In-Reply-To: Message-Id: <308670DB-BD2A-4738-81B1-3F6FB312C0C8@apache.org> X-Mailer: Apple Mail (2.3273) Operators and hooks don't need any special plugin system - simply having = them as as separate Python modules which are imported using normal = python semantics is enough. In fact now that I think about it: I want to deprecate the plugins = registering hooks/operators etc and limit it to only bits which a simple = python import can't manage - which I think is only anything that needs = to be registered with another system, such as custom routes in the web = UI. I'll draft an AIP for this soon. -ash > On 18 Sep 2018, at 00:50, George Leslie-Waksman = wrote: >=20 > Given we have a plugin system, could we alternatively move away from > keeping non-core supported code outside of the core project/repo? >=20 > It would hugely decrease the surface area of the main repository and > testing infrastructure to get most of the contrib code out to its own = place. >=20 > Further, it would decrease the committer burden of having to = approve/merge > code that is not supposed to be their responsibility. >=20 > On Mon, Sep 17, 2018 at 4:37 PM Tim Swast = wrote: >=20 >>> Individual operators and hooks living in separate repositories on = github >> (or possibly other Apache projects), which are then distributed by = pip and >> installed as libraries seems like it would scale better. >>=20 >> Pandas did this about a year ago, and it's seemed to have worked = well. For >> example, pandas.read_gbq is a very thin wrapper around = pandas_gbq.read_gbq >> (distributed as a separate package). It has made it easier for me to = track >> issues corresponding to my area of expertise. >>=20 >> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan = wrote: >>=20 >>>> My understanding as a contributor is that if a hook/operator is in >> core, >>> it >>>> means that a committer is willing to take personal responsibility = to >>>> maintain it (or at least help maintain it), and everything else = goes in >>>> contrib. >>>=20 >>> That's not correct. All of the code is owned by the entire >>> community[1]; no one person is responsible for any of it. There's = no >>> silos, fiefdoms, walled gardens, etc. If the community cannot = support >>> a piece of code it should be deprecated and subsequently removed. >>>=20 >>> Contrib sections are almost always problematic for this reason. >>> Hadoop ended up abandoning its. Because Airflow acts as a gathering >>> point for so many disparate technologies (databases, storage = systems, >>> compute engines, etc.), trying to keep all of them corralled and up = to >>> date will be very difficult. Individual operators and hooks living = in >>> separate repositories on github (or possibly other Apache projects), >>> which are then distributed by pip and installed as libraries seems >>> like it would scale better. >>>=20 >>> -Jakob >>>=20 >>> [1] = https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie >>>=20 >>> On 15 September 2018 at 13:29, Jeff Payne = wrote: >>>> How many operators are added to contrib per month? Is it too many = to >>> make the decision case by case? If so, then the above mentioned rule >> sounds >>> fairly reasonable. However, if that's the rule, shouldn't a bunch of >>> existing modules be moved from contrib to core? >>>>=20 >>>> Get Outlook for Android >>>>=20 >>>> ________________________________ >>>> From: Taylor Edmiston >>>> Sent: Saturday, September 15, 2018 1:13:47 PM >>>> To: dev@airflow.incubator.apache.org >>>> Subject: Re: Guidelines on Contrib vs Non-contrib >>>>=20 >>>> My understanding as a contributor is that if a hook/operator is in >> core, >>> it >>>> means that a committer is willing to take personal responsibility = to >>>> maintain it (or at least help maintain it), and everything else = goes in >>>> contrib. >>>>=20 >>>> *Taylor Edmiston* >>>> Blog | LinkedIn >>>> | Stack Overflow >>>> | = Developer >>> Story >>>> >>>>=20 >>>>=20 >>>>=20 >>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik >> wrote: >>>>=20 >>>>> Hi, all (mainly contributors), >>>>>=20 >>>>> Can we decide on a common guideline on when a hook/operator should = go >>> under >>>>> contrib vs core? >>>>>=20 >>>>> Regards, >>>>>=20 >>>>> *Kaxil Naik* >>>>> *Big Data Consultant *@ *Data Reply UK* >>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark = & >>> Neo4j >>>>> Developer >>>>> *Phone: *+44 (0) 74820 88992 >>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >>>>>=20 >>>=20 >> -- >> * =E2=80=A2 **Tim Swast* >> * =E2=80=A2 *Software Friendliness Engineer >> * =E2=80=A2 *Google Cloud Developer Relations >> * =E2=80=A2 *Seattle, WA, USA >>=20