From dev-return-7612-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Mon Feb 11 17:14:20 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8BC97180648 for ; Mon, 11 Feb 2019 18:14:19 +0100 (CET) Received: (qmail 13136 invoked by uid 500); 11 Feb 2019 17:14:18 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 13104 invoked by uid 99); 11 Feb 2019 17:14:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Feb 2019 17:14:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 53ED2C224D for ; Mon, 11 Feb 2019 17:14:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.549 X-Spam-Level: ** X-Spam-Status: No, score=2.549 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id bxzuUGtoh21r for ; Mon, 11 Feb 2019 17:14:15 +0000 (UTC) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id EF6275F206 for ; Mon, 11 Feb 2019 17:14:14 +0000 (UTC) Received: by mail-lj1-f173.google.com with SMTP id v14-v6so9493333ljv.1 for ; Mon, 11 Feb 2019 09:14:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c2umpKCZt+8hblbItF80JxMOFLbF6Er0KH+k7O+Nk9o=; b=Gj6gQ+dcxQH66TmeS0Ouz8EVJd6J/X0rmTYwIP7v9QLYlqKEHALwqmry/Q407sQ9a5 L+B5wCjCDqNhF70qAjxtTg56AN2Bo4FEgbA4WWv0JYRm72mqmsHVDii+zvrZFXvIZ6JR 3OcxU+QrDGxXy6vTVO73XZD5qbs6MRMQw6M+FbeiHVLiETG0rf8UXcoVW4UT76fYAhlS 4gY64XzFUu66pcp7G+ym4ItGE1D7NS6OphedoKBwM85crSCsO20l/o8GIAaaasW8MNp1 oCuBei+BMFZdieXhq9P0mDzil0CcLEV4AK5MFOZXqsLzj+UbG79YfEHpUTW1ds97dxuV vdrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c2umpKCZt+8hblbItF80JxMOFLbF6Er0KH+k7O+Nk9o=; b=hPvIiYqGzN1+3tZ+GnG3pBNxuEAsbddc9Uuyjj1JHwjnwT1UoCOvhRznYwZpyrCgbI g+Sr2elueR8KfCWkiM363TceuK6+JkxJmSJMkK0Dyes/2cp9xcM24CVI7vnH3NUWFAdt ONzbbNdA/5a540HQCiOk+hqL/Ucgtg1YKa7vsU0R0YTuXWD76thJdG1O9Bt2u3xprKt3 NQE71wA8wq4eMyVAvF5c7fc+7wWnLkUK1Yyx5XKenfxfDsWLt3vP7+cnTHc5grI5z2Ai bJjUuKn8sV54WYAtQz+K30wkNZvZQD3mfhCzZ6aqX2Y/1098Pof/06Zbw7aQ1uk9Bm+W +jTQ== X-Gm-Message-State: AHQUAubDrFUjDWdugfY6K+J8Aylz6jwoEAU3Gz7xvm/VjgPtMAwm3Raz cvnwRmiyD6tCT0B36/yFWcjlo7fr23IR6y/2fEj1Mg== X-Google-Smtp-Source: AHgI3Ia+cnnEy00DWSVi4aoDMLVgqmE4oAEV+HZsrhNStBymE9fJiiGn4ESSnU6i8QhpBCh0haTCY+vdGEu+epFVIdg= X-Received: by 2002:a2e:7e11:: with SMTP id z17-v6mr14918545ljc.106.1549905253067; Mon, 11 Feb 2019 09:14:13 -0800 (PST) MIME-Version: 1.0 References: <3F133B08-5EA9-47CD-A4F4-FFF0B263C7AE@gmail.com> In-Reply-To: From: Maxime Beauchemin Date: Mon, 11 Feb 2019 09:14:01 -0800 Message-ID: Subject: Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0? To: dev@airflow.apache.org Cc: airflowuser Content-Type: multipart/alternative; boundary="0000000000000b737c0581a17065" --0000000000000b737c0581a17065 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable From memory, I think MesosExecutor depends on pickling to get DAG definitions to workers, which we should also deprecate. About CeleryExecutor, we never had the intention to make it the recommended option for production early on. The intent back in 2014 was to write a YarnExecutor quickly (that was before the rise of k8s), but realized that container support wasn't ready in Yarn at the time, and had hard restrictions on what we could install on Hadoop/Yarn data nodes, so we moved forward with Celery. I think it's practical to have LocalExecutor around though for unit test, local use, and for POCs. Max On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < o.girardot@lateral-thoughts.com> wrote: > Hi, we tried to use it for a new project 1year ago, but it was too broken > to be usable, and we didn't have the time to make it work. > Of course we are still hoping to use it one day, but that doesn't mean it > should not move to contrib, it was already clear from the documentation > that it was not a first class citizen. > > Regards, > > Olivier. > > Le dim. 27 janv. 2019 =C3=A0 12:12, Ash Berlin-Taylor a = =C3=A9crit > : > > > That PR pre-dates 1.9.0 being released. > > > > > On 27 Jan 2019, at 06:57, airflowuser .INVALID> > > wrote: > > > > > > The Mesos executor is being used (at least by some people) and there = is > > an open PR to extend it: > > > https://github.com/apache/airflow/pull/3739 > > > > > > > > > Sent with ProtonMail Secure Email. > > > > > > =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 Origi= nal Message =E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90=E2=80=90 > > > On Friday, January 25, 2019 7:30 PM, Kaxil Naik > > wrote: > > > > > >> What about Dask Executor as well. > > >> > > >> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > daniel.imberman@gmail.com > > >> wrote: > > >> > > >>> @sid basically my response lol > > >>> I haven't seen any emails from any users and if it's been broken fo= r > > two > > >>> versions then there doesn't seem to be enough interest to maintain > it. > > >>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote: > > >>> > > >>>> LOL :_0 > > >>>> -s > > >>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com > > >>>> wrote: > > >>>> > > >>>>> I agree. Maybe we should make a choice on what we consider 1st > class > > >>>>> executors and move others to contrib (I'm looking at you celery) > > >>>>> Sent from my iPhone > > >>>>> > > >>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org wrote= : > > >>>>>> Is anyone using the Mesos Executor? I think we should deprecate > and > > >>>>>> remove it. > > >>>>>> The reason I say that is I have a feeling it's been broken since > > >>>>>> 1.9.0 > > >>>> > > >>>>> and no-one has noticed: > > >>>>> > > >>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > > >>>>>> def start(self): > > >>>>>> # ... > > >>>>>> self.log.info( > > >>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > > >>>>>> %s, > > >>>> > > >>>>> checkpoint : %s', > > >>>>> > > >>>>>> master, framework.name, > > >>>>>> str(task_cpu), str(task_memory), > str(framework.checkpoint) > > >>>>>> ) > > >>>>>> > > >>>>>> > > >>>>>> That mixin should have been LoggingMixin, not LoginMixin. But > given > > >>>>>> this > > >>>>>> bug has been in since > > >>> > > >>> > > > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee69= 1f181b29 > > >>> > > >>>>> which was in the 1.9.0 release I propose we simply remove this > > executor > > >>>>> from master. > > >>>>> > > >>>>>> Thoughts? > > >>>>>> Ash > > > > > > > > > > > --0000000000000b737c0581a17065--