From dev-return-8130-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Fri Apr 12 21:43:00 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6C6DC180621 for ; Fri, 12 Apr 2019 23:43:00 +0200 (CEST) Received: (qmail 76389 invoked by uid 500); 12 Apr 2019 21:42:59 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 76377 invoked by uid 99); 12 Apr 2019 21:42:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Apr 2019 21:42:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C0815C0DA0 for ; Fri, 12 Apr 2019 21:42:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.542 X-Spam-Level: ** X-Spam-Status: No, score=2.542 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.258, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 7FICc3NPIQao for ; Fri, 12 Apr 2019 21:42:56 +0000 (UTC) Received: from mail-it1-f193.google.com (mail-it1-f193.google.com [209.85.166.193]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 7CE5A60D34 for ; Fri, 12 Apr 2019 21:42:56 +0000 (UTC) Received: by mail-it1-f193.google.com with SMTP id v8so1473583itf.0 for ; Fri, 12 Apr 2019 14:42:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=UZ9IM19gMzPvocZ53inaKTATODNQxHUZZ/959J0DJgM=; b=LW9woVTwRPx7Ch7s27PfG7SFmPcFFF1SG2fDYaIlFGgRWOkH9zfKRtYczbbuIrvoJm XwcxB7VUrD28sMTYAvnXAFT2J6ukFEjrXldnUyOzB6Z9RYPr7HkUFpIKKE0b2jbtDb6B Y+2SiUnYkfKSCbClkmyPg0JFgvYgRYL2oa6wYARlk18Gaued/ISKv8z8NUZ2AZXaf9gd v6xjI9umTJO8E8w7dDNjH7mBOYRNYlAJmcjEympffcRBuk8OrjRM4nPdfKXorT1nBB7w xxxLJKQu2XjQnMioDJ6eT8hZ7EmAliXdMNoCHV3Ts1msytRBYnhH/FLzVzy5160+JGf/ aHAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=UZ9IM19gMzPvocZ53inaKTATODNQxHUZZ/959J0DJgM=; b=iQYJC6LYQBBS+mtM5zDKn6hsFekpyOD25wSMGi9oSDwMJcZ7ycqJIMlp9lZ5nvjbqA AxRRlXvf8Fhi/4K/zgRsu7O/S8TIVan+/H9GNZ3uLYTmM3h97fe3p60KDDr8tfsHHLk2 s0J3iQII2VEptKoJMdEYZxYz/hMxiSVZLFTj6Ah0OQY7l+Y3Nbc+EnNWDGf54aMixk1m AD+SZRpRhMSV/P4Rwl9xAYFqrawkuT4JdMIFMTLKgXdi8e8EoOs7OqM9e/ddOzhozzpv jeIgrINhiMfs2pOMjScXWauMFlttahARYix1KkrU89HNjR5tdtAwTjfCVg1RGxrPG7+T JZig== X-Gm-Message-State: APjAAAWY91Jh2UV7eGMAp3axDk/4gi8jnyyut7Tb8xzzB0DyL+8Ciw71 ognDZXDlkM7h1of5iKVuXM4nsB832ZAVq/gMGk/MTg== X-Google-Smtp-Source: APXvYqxg9RvtEXN6IaXP7/yySeo3svW0KDNnn1cZtZApyE/+JH/n/EaIshAtgHTup3X3G8AZed5ndcxrv+KU3cedBr0= X-Received: by 2002:a24:920a:: with SMTP id l10mr14388198itd.98.1555105375314; Fri, 12 Apr 2019 14:42:55 -0700 (PDT) MIME-Version: 1.0 References: <3F133B08-5EA9-47CD-A4F4-FFF0B263C7AE@gmail.com> <9895F44D-7D0B-4116-91C8-DEA935846545@apache.org> In-Reply-To: From: Tao Feng Date: Fri, 12 Apr 2019 14:42:18 -0700 Message-ID: Subject: Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0? To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="0000000000007be85805865c2f0a" --0000000000007be85805865c2f0a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable +1 on removing mesos executor. On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman wrote: > Since we are now doing version 2.0 spring cleaning can we revisit this? > > On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin < > maximebeauchemin@gmail.com> wrote: > > > +1 as it relies on pickles > > > > On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor > wrote: > > > > > I say lets remove it in 2.0. > > > > > > > On 27 Feb 2019, at 11:50, Driesprong, Fokko > > > wrote: > > > > > > > > Are we going to remove the MesosExecutor? I think it is a safe bet > > since > > > in > > > > the Survey of Ash (thanks again for that!), it also shows that nobo= dy > > is > > > > really using it. > > > > > > > > https://jira.apache.org/jira/browse/AIRFLOW-3969 > > > > > > > > Cheers, Fokko > > > > > > > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor < > ash@apache.org > > >: > > > > > > > >> I=E2=80=99m glad yarn wasn=E2=80=99t the only option - it would ha= ve meant I=E2=80=99d have > > > never > > > >> been in a position to use Airflow! (Many of our workflows don=E2= =80=99t > touch > > > >> EMR/Hadoop, and running Celery is much more of a known element to = a > > > python > > > >> developer than configuring Yarn) > > > >> > > > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin < > > > maximebeauchemin@gmail.com> > > > >> wrote: > > > >>> > > > >>> From memory, I think MesosExecutor depends on pickling to get DAG > > > >>> definitions to workers, which we should also deprecate. > > > >>> > > > >>> About CeleryExecutor, we never had the intention to make it the > > > >> recommended > > > >>> option for production early on. The intent back in 2014 was to > write > > a > > > >>> YarnExecutor quickly (that was before the rise of k8s), but > realized > > > that > > > >>> container support wasn't ready in Yarn at the time, and had hard > > > >>> restrictions on what we could install on Hadoop/Yarn data nodes, = so > > we > > > >>> moved forward with Celery. > > > >>> > > > >>> I think it's practical to have LocalExecutor around though for un= it > > > test, > > > >>> local use, and for POCs. > > > >>> > > > >>> Max > > > >>> > > > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > > > >>> o.girardot@lateral-thoughts.com> wrote: > > > >>> > > > >>>> Hi, we tried to use it for a new project 1year ago, but it was t= oo > > > >> broken > > > >>>> to be usable, and we didn't have the time to make it work. > > > >>>> Of course we are still hoping to use it one day, but that doesn'= t > > mean > > > >> it > > > >>>> should not move to contrib, it was already clear from the > > > documentation > > > >>>> that it was not a first class citizen. > > > >>>> > > > >>>> Regards, > > > >>>> > > > >>>> Olivier. > > > >>>> > > > >>>> Le dim. 27 janv. 2019 =C3=A0 12:12, Ash Berlin-Taylor > a > > > >> =C3=A9crit > > > >>>> : > > > >>>> > > > >>>>> That PR pre-dates 1.9.0 being released. > > > >>>>> > > > >>>>>> On 27 Jan 2019, at 06:57, airflowuser < > airflowuser@protonmail.com > > > >>>> .INVALID> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>> The Mesos executor is being used (at least by some people) and > > there > > > >> is > > > >>>>> an open PR to extend it: > > > >>>>>> https://github.com/apache/airflow/pull/3739 > > > >>>>>> > > > >>>>>> > > > >>>>>> Sent with ProtonMail Secure Email. > > > >>>>>> > > > >>>>>> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80= =90 Original Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90= =E2=80=90 > > > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik < > > > kaxilnaik@gmail.com> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>>> What about Dask Executor as well. > > > >>>>>>> > > > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > > > >>>> daniel.imberman@gmail.com > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> @sid basically my response lol > > > >>>>>>>> I haven't seen any emails from any users and if it's been > broken > > > for > > > >>>>> two > > > >>>>>>>> versions then there doesn't seem to be enough interest to > > maintain > > > >>>> it. > > > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com > > wrote: > > > >>>>>>>>> > > > >>>>>>>>> LOL :_0 > > > >>>>>>>>> -s > > > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin > > bdbruin@gmail.com > > > >>>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> I agree. Maybe we should make a choice on what we consider > 1st > > > >>>> class > > > >>>>>>>>>> executors and move others to contrib (I'm looking at you > > celery) > > > >>>>>>>>>> Sent from my iPhone > > > >>>>>>>>>> > > > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.or= g > > > >> wrote: > > > >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should > > deprecate > > > >>>> and > > > >>>>>>>>>>> remove it. > > > >>>>>>>>>>> The reason I say that is I have a feeling it's been broke= n > > > since > > > >>>>>>>>>>> 1.9.0 > > > >>>>>>>>> > > > >>>>>>>>>> and no-one has noticed: > > > >>>>>>>>>> > > > >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > > > >>>>>>>>>>> def start(self): > > > >>>>>>>>>>> # ... > > > >>>>>>>>>>> self.log.info( > > > >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > > > >>>>>>>>>>> %s, > > > >>>>>>>>> > > > >>>>>>>>>> checkpoint : %s', > > > >>>>>>>>>> > > > >>>>>>>>>>> master, framework.name, > > > >>>>>>>>>>> str(task_cpu), str(task_memory), > > > >>>> str(framework.checkpoint) > > > >>>>>>>>>>> ) > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. > But > > > >>>> given > > > >>>>>>>>>>> this > > > >>>>>>>>>>> bug has been in since > > > >>>>>>>> > > > >>>>>>>> > > > >>>>> > > > >>>> > > > >> > > > > > > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee69= 1f181b29 > > > >>>>>>>> > > > >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove > this > > > >>>>> executor > > > >>>>>>>>>> from master. > > > >>>>>>>>>> > > > >>>>>>>>>>> Thoughts? > > > >>>>>>>>>>> Ash > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>> > > > >> > > > >> > > > > > > > > > --0000000000007be85805865c2f0a--