From dev-return-8146-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Sat Apr 13 10:36:00 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9B29F180668 for ; Sat, 13 Apr 2019 12:36:00 +0200 (CEST) Received: (qmail 78778 invoked by uid 500); 13 Apr 2019 10:35:59 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 78765 invoked by uid 99); 13 Apr 2019 10:35:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Apr 2019 10:35:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D822FC6416 for ; Sat, 13 Apr 2019 10:35:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=polidea.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 3vvvQumsiCTf for ; Sat, 13 Apr 2019 10:35:55 +0000 (UTC) Received: from mail-it1-f172.google.com (mail-it1-f172.google.com [209.85.166.172]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B99045F529 for ; Sat, 13 Apr 2019 10:35:54 +0000 (UTC) Received: by mail-it1-f172.google.com with SMTP id v8so2493698itf.0 for ; Sat, 13 Apr 2019 03:35:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=polidea.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=TNLqUSRW5jh9TM777QQyOuZrnjcmM9FGbvTVQqqjhJI=; b=yboKZxelHGJLY4k35bo3in90Z+Zov8bqgCy+ok+MhfjM+oWkebH/9o/2PkZg0dc4K6 LyvZC0z6+/CfDTryq5wkrPeV4XuVGRk4jvpgrEB3fP1qbFtC0i6zzBZhJjbygAlUcTrI wyrT0QA7KKq7GMH+tOD6z8BIOjj4Yix5MgILo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=TNLqUSRW5jh9TM777QQyOuZrnjcmM9FGbvTVQqqjhJI=; b=I16vXbeGpej0ljsbF1reyQ302fB+6TY3fGvKCGfpcmMjwujCaXbTr2ihTGpnjBjR1i hBKORSshiyVkIeWIC0ZNX9mwLyhHxHIaTDkbon++J98KDpgN2dPOR47TkeLCH0R5L9vo n98n83i8HJP8/8ZmrTzJ1zbijxTM2teyM6dhkYrupRV8i3DWHnOv/6eD/OJGVKi4C918 Spn2IAR8ppO1PDG7yq1vgUC68/kLT6IXJHbhtL8rwvkoRRXo82TVgHHDip50oXvPgw4i DzqgVh9P5ptokuCq5idtVchZBKdg//Ziq2y8QBHUIQ05sAQ1znMWr7XQgCdvt5E+xQuX rFAQ== X-Gm-Message-State: APjAAAVBrjdv9WqEWnYTIdnm8IXv1dchqCNlZzqssJLVr7Y+vH2Jn9St nK4GH9OE/F8yCbdoXJqpVWAVNSGFkVTGJaQkOSe+Wnv7 X-Google-Smtp-Source: APXvYqwDiNukKBkU1y8k3wOdn7UgqTVz8/2f/BhLMviHv5wo59MP8rDbBl3LSNbm8PQeAwhXuph4MoElTRf8fTzD51o= X-Received: by 2002:a24:628b:: with SMTP id d133mr16422600itc.32.1555151753089; Sat, 13 Apr 2019 03:35:53 -0700 (PDT) MIME-Version: 1.0 References: <24F53EDA-5860-4B6B-A305-8DA09672A1FA@gmail.com> <6722A60B-8225-4A65-9786-FCA3A0C515D5@gmail.com> <4fa41ee6-f6b6-acfa-80fb-16d5fce13e27@gmx.de> In-Reply-To: <4fa41ee6-f6b6-acfa-80fb-16d5fce13e27@gmx.de> From: Jarek Potiuk Date: Sat, 13 Apr 2019 12:35:41 +0200 Message-ID: Subject: Re: [2.0 spring cleaning] Deprecate contrib folder? To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000d0acc0058666fb03" --000000000000d0acc0058666fb03 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I think there are quite a few contrib parts that are at least on-par with regards to code quality, testing and especially documentation. And yes, among those are GCP operators we developed which are not only unit but also system-tested and we put quite an effort into making documentation really useful and well structured ;). I'd rather move those "graduated" operators/hooks to core and maybe rename the "contrib" folder to "incubator" or something like that to indicate that those operators are not yet "core-quality" but aspire to become one. That would make a nice "intro" task for new contributors - to improve one of the incubating operators to become "core-ready". I am quite sceptical myself about AIP-8 and separating out the hooks and operators. There were already few discussions about that, but splitting the operators out might be quite difficult and it will only be possible if there is some way to quickly test compatibility of those split operators with various versions of Airflow and set of dependent requirements.. Otherwise it will very quickly become a mess - nobody will know which version of Airflow is needed to run which operators and there will be problems if someone will try to run different operators with different requirements in the same DAG (and different versions of airflow). Until we manage to isolate operators within the same DAG to potentially use different dependencies, this is straight road to dependency-hell. One solution to that that I have in mind for some time (but this is very long term) might be to make Airflow Docker-native and run every operator within it's own separate Docker instance with it's own dependencies. That would be quite possible to do (we would need to split operators into very light "proxy" (basically current __init__() - the part that is executed within DAG scanning) and heavy "execute" parts (where the operator's execute-related methods would be run in separate Docker on workers). J. On Sat, Apr 13, 2019 at 11:59 AM Felix Uellendall wrote: > +1 on deprecating the contrib folder. > > Bolk de Bruin the reason the core hooks and operators are properly > tested because, for example I added some more tests to it and I am > "only" a contributor. > > So do you really want to split up contributors work and core committers > work? I personally think this is not the right way to go. > > It is true that the contrib hooks and operators have not the same level > of code quality but we can do something about it. I am trying to improve > our test coverage overall and add missing tests. > > I don't think an extra step is needed here where we first move properly > tested ones into the core package and then moving new ones from time to > time into it. Wouldn't that mean that we think the code quality of > "contrib" (contributor) in general is worse than the code quality of > committers? Every new contributor who comes along this project would > think that, wouldn't he? > > -feluelle > > Am 13/04/2019 um 07:51 schrieb Beau Barker: > > A separate airflow-contrib repo, on a separate release cadence would be > my preference. > > > > > >> On 12 Apr 2019, at 11:17 pm, Julian De Ruiter < > julianderuiter@godatadriven.com> wrote: > >> > >> Isn=E2=80=99t this in contradiction with AIP-8, which is aimed at remo= ving > operators/hooks from the core Airflow package? > >> > >> Personally I would rather remove hooks/operators from Airflow than add > even more to the Airflow core. This counts double for the contrib stuff, > which is often poorly designed and/or tested. > >> > >> Best, > >> Julian > >> > >>> On 12 Apr 2019, at 10:23, Bolke de Bruin wrote: > >>> > >>> That=E2=80=99s perfectly fine to me. > >>> > >>> Verstuurd vanaf mijn iPad > >>> > >>>> Op 12 apr. 2019 om 10:20 heeft Kaxil Naik het > volgende geschreven: > >>>> > >>>> Ok. How about moving the properly tested and maintained hooks/ops fr= om > >>>> contrib to core? > >>>> > >>>>> On Fri, Apr 12, 2019, 09:13 Bolke de Bruin > wrote: > >>>>> > >>>>> I disagree. Core signals =E2=80=9Cproperly tested=E2=80=9D and main= tained. Ie. A > kind of > >>>>> quality. I don=E2=80=99t think contrib has that. > >>>>> > >>>>> Verstuurd vanaf mijn iPad > >>>>> > >>>>>> Op 12 apr. 2019 om 10:03 heeft Kaxil Naik he= t > >>>>> volgende geschreven: > >>>>>> Contrib folder was used when it was used at Airbnb. Currently, it > doesn't > >>>>>> make any sense and we have equal responsibility to maintain all th= e > >>>>> hooks, > >>>>>> operators, sensors in contrib folder as we do for core. > >>>>>> > >>>>>> I would suggest to remove contrib folder and move all hooks, ops, > and > >>>>>> sensors to the core folder. > >>>>>> > >>>>>> Or reorganize the folder structure similar to what was discussed i= n > a > >>>>>> mailing thread few months ago. > >>>>>> > >>>>>> Regards, > >>>>>> Kaxil > --=20 Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129 <+48660796129> E: jarek.potiuk@polidea.com --000000000000d0acc0058666fb03--