From dev-return-7703-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Wed Feb 27 11:58:53 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D0D80180608 for ; Wed, 27 Feb 2019 12:58:52 +0100 (CET) Received: (qmail 12045 invoked by uid 500); 27 Feb 2019 11:58:51 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 12019 invoked by uid 99); 27 Feb 2019 11:58:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2019 11:58:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 823FFC8B22 for ; Wed, 27 Feb 2019 11:58:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.751 X-Spam-Level: ** X-Spam-Status: No, score=2.751 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=driesprong-frl.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id PGOlWlcxMpOi for ; Wed, 27 Feb 2019 11:58:47 +0000 (UTC) Received: from mail-yw1-f42.google.com (mail-yw1-f42.google.com [209.85.161.42]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 21EC85F645 for ; Wed, 27 Feb 2019 11:51:13 +0000 (UTC) Received: by mail-yw1-f42.google.com with SMTP id k14so7838841ywe.4 for ; Wed, 27 Feb 2019 03:51:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=driesprong-frl.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Do/iRygvtMgmrxE25uBWqrD3BKbXQDGBP3+U1zsXuFQ=; b=Lw/PkVcw1T7+TYjCzyk3xatYMZHDFZgBC9cbWPIPWMI4AHjsA3uJdxtWMYt7GyQk0Z sKe6mxrdlcc8NJJb2n7Z3urh1a+7IhtMtCi83V3eYUVcbw6mj+eRq2XZLmlF0v62FU8+ WPOzBuuBXVABv2tqp2q/SPm/oih7LV3namw4r1YFJzytYW7Vlzx5pEJR18a4zaLspTiL 2/H9b4bf+QP3M4oI/knQ7nBVC3JPHtJITnqzgsdaSNz9xnkjZamc3OpARYQaHHdBS2pf ZWDIWZjM/sFhmYtYrKOidM4wMDU7+DmPeOJmqre5zo3x72H3Po/FVqs3odr8wNK0Vfh/ Jv/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Do/iRygvtMgmrxE25uBWqrD3BKbXQDGBP3+U1zsXuFQ=; b=aXxsMpiLlBG0ZyLjG/HBY9PUlX3hNhY3oKsUgZjfV9pgy7h4L16Z5UmUVjnKmcfaUA rIi3xYUnW27fB7+rNjOCWfT7bBjtHKSrOiGuJgZab3R+6TxBDArN6cdGOYpMKiSBYqiQ Z/DI4I6PfyjNrJQLX9H8VRDyi8KDFbvg44GnEmTx1s3USaMzHVeTDmN/nD/zyL3RyMrr kGRzjCObfDSEGH/6v2KsRs9fbuRPg7YkdYRtkUd/I7k9wb+1Ou+EtzYHnqobLLOctAit 3jTQkkiP7hBa3p9zwBoYCZR0YnZ2J/fCHFiG7H6XriN7lWK0V6PIlzY1baUCoUHLSe4g YMOw== X-Gm-Message-State: AHQUAubdsijGaM/VTqbDk9WgWleHs8qZBjX1e6NF5lq3mKbfWfmRU57+ H8VxSLcL05R54bAiNec0P6/60s/A+8IrRSMt83foOqlrx5OANA== X-Google-Smtp-Source: AHgI3IZ7aEjJUKbv88UXPAtKLfeUfa8xbcB5NHsWBe2KsIKAzrA98YSuqCZWxl6GwAQnBXJ8bRpxQkbzrQMXgeHLZrk= X-Received: by 2002:a81:7092:: with SMTP id l140mr658536ywc.316.1551268266580; Wed, 27 Feb 2019 03:51:06 -0800 (PST) MIME-Version: 1.0 References: <3F133B08-5EA9-47CD-A4F4-FFF0B263C7AE@gmail.com> <9895F44D-7D0B-4116-91C8-DEA935846545@apache.org> In-Reply-To: <9895F44D-7D0B-4116-91C8-DEA935846545@apache.org> From: "Driesprong, Fokko" Date: Wed, 27 Feb 2019 12:50:55 +0100 Message-ID: Subject: Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0? To: dev@airflow.apache.org Content-Type: multipart/alternative; boundary="000000000000fb34090582dec975" --000000000000fb34090582dec975 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Are we going to remove the MesosExecutor? I think it is a safe bet since in the Survey of Ash (thanks again for that!), it also shows that nobody is really using it. https://jira.apache.org/jira/browse/AIRFLOW-3969 Cheers, Fokko Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor : > I=E2=80=99m glad yarn wasn=E2=80=99t the only option - it would have mean= t I=E2=80=99d have never > been in a position to use Airflow! (Many of our workflows don=E2=80=99t t= ouch > EMR/Hadoop, and running Celery is much more of a known element to a pytho= n > developer than configuring Yarn) > > > On 11 Feb 2019, at 17:14, Maxime Beauchemin > wrote: > > > > From memory, I think MesosExecutor depends on pickling to get DAG > > definitions to workers, which we should also deprecate. > > > > About CeleryExecutor, we never had the intention to make it the > recommended > > option for production early on. The intent back in 2014 was to write a > > YarnExecutor quickly (that was before the rise of k8s), but realized th= at > > container support wasn't ready in Yarn at the time, and had hard > > restrictions on what we could install on Hadoop/Yarn data nodes, so we > > moved forward with Celery. > > > > I think it's practical to have LocalExecutor around though for unit tes= t, > > local use, and for POCs. > > > > Max > > > > On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > > o.girardot@lateral-thoughts.com> wrote: > > > >> Hi, we tried to use it for a new project 1year ago, but it was too > broken > >> to be usable, and we didn't have the time to make it work. > >> Of course we are still hoping to use it one day, but that doesn't mean > it > >> should not move to contrib, it was already clear from the documentatio= n > >> that it was not a first class citizen. > >> > >> Regards, > >> > >> Olivier. > >> > >> Le dim. 27 janv. 2019 =C3=A0 12:12, Ash Berlin-Taylor = a > =C3=A9crit > >> : > >> > >>> That PR pre-dates 1.9.0 being released. > >>> > >>>> On 27 Jan 2019, at 06:57, airflowuser >> .INVALID> > >>> wrote: > >>>> > >>>> The Mesos executor is being used (at least by some people) and there > is > >>> an open PR to extend it: > >>>> https://github.com/apache/airflow/pull/3739 > >>>> > >>>> > >>>> Sent with ProtonMail Secure Email. > >>>> > >>>> =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Orig= inal Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80= =90 > >>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik > >>> wrote: > >>>> > >>>>> What about Dask Executor as well. > >>>>> > >>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > >> daniel.imberman@gmail.com > >>>>> wrote: > >>>>> > >>>>>> @sid basically my response lol > >>>>>> I haven't seen any emails from any users and if it's been broken f= or > >>> two > >>>>>> versions then there doesn't seem to be enough interest to maintain > >> it. > >>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote: > >>>>>>> > >>>>>>> LOL :_0 > >>>>>>> -s > >>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com > >>>>>>> wrote: > >>>>>>> > >>>>>>>> I agree. Maybe we should make a choice on what we consider 1st > >> class > >>>>>>>> executors and move others to contrib (I'm looking at you celery) > >>>>>>>> Sent from my iPhone > >>>>>>>> > >>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org > wrote: > >>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate > >> and > >>>>>>>>> remove it. > >>>>>>>>> The reason I say that is I have a feeling it's been broken sinc= e > >>>>>>>>> 1.9.0 > >>>>>>> > >>>>>>>> and no-one has noticed: > >>>>>>>> > >>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > >>>>>>>>> def start(self): > >>>>>>>>> # ... > >>>>>>>>> self.log.info( > >>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > >>>>>>>>> %s, > >>>>>>> > >>>>>>>> checkpoint : %s', > >>>>>>>> > >>>>>>>>> master, framework.name, > >>>>>>>>> str(task_cpu), str(task_memory), > >> str(framework.checkpoint) > >>>>>>>>> ) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But > >> given > >>>>>>>>> this > >>>>>>>>> bug has been in since > >>>>>> > >>>>>> > >>> > >> > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee69= 1f181b29 > >>>>>> > >>>>>>>> which was in the 1.9.0 release I propose we simply remove this > >>> executor > >>>>>>>> from master. > >>>>>>>> > >>>>>>>>> Thoughts? > >>>>>>>>> Ash > >>>> > >>>> > >>> > >>> > >> > > --000000000000fb34090582dec975--