airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Feng <fengta...@gmail.com>
Subject Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Date Fri, 12 Apr 2019 21:42:18 GMT
+1 on removing mesos executor.

On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman <daniel.imberman@gmail.com>
wrote:

> Since we are now doing version 2.0 spring cleaning can we revisit this?
>
> On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > +1 as it relies on pickles
> >
> > On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <ash@apache.org>
> wrote:
> >
> > > I say lets remove it in 2.0.
> > >
> > > > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fokko@driesprong.frl>
> > > wrote:
> > > >
> > > > Are we going to remove the MesosExecutor? I think it is a safe bet
> > since
> > > in
> > > > the Survey of Ash (thanks again for that!), it also shows that nobody
> > is
> > > > really using it.
> > > >
> > > > https://jira.apache.org/jira/browse/AIRFLOW-3969
> > > >
> > > > Cheers, Fokko
> > > >
> > > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <
> ash@apache.org
> > >:
> > > >
> > > >> I’m glad yarn wasn’t the only option - it would have meant I’d
have
> > > never
> > > >> been in a position to use Airflow! (Many of our workflows don’t
> touch
> > > >> EMR/Hadoop, and running Celery is much more of a known element to
a
> > > python
> > > >> developer than configuring Yarn)
> > > >>
> > > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> From memory, I think MesosExecutor depends on pickling to get
DAG
> > > >>> definitions to workers, which we should also deprecate.
> > > >>>
> > > >>> About CeleryExecutor, we never had the intention to make it the
> > > >> recommended
> > > >>> option for production early on. The intent back in 2014 was to
> write
> > a
> > > >>> YarnExecutor quickly (that was before the rise of k8s), but
> realized
> > > that
> > > >>> container support wasn't ready in Yarn at the time, and had hard
> > > >>> restrictions on what we could install on Hadoop/Yarn data nodes,
so
> > we
> > > >>> moved forward with Celery.
> > > >>>
> > > >>> I think it's practical to have LocalExecutor around though for
unit
> > > test,
> > > >>> local use, and for POCs.
> > > >>>
> > > >>> Max
> > > >>>
> > > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> > > >>> o.girardot@lateral-thoughts.com> wrote:
> > > >>>
> > > >>>> Hi, we tried to use it for a new project 1year ago, but it
was too
> > > >> broken
> > > >>>> to  be usable, and we didn't have the time to make it work.
> > > >>>> Of course we are still hoping to use it one day, but that
doesn't
> > mean
> > > >> it
> > > >>>> should not move to contrib, it was already clear from the
> > > documentation
> > > >>>> that it was not a first class citizen.
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Olivier.
> > > >>>>
> > > >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <ash@apache.org>
> a
> > > >> écrit
> > > >>>> :
> > > >>>>
> > > >>>>> That PR pre-dates 1.9.0 being released.
> > > >>>>>
> > > >>>>>> On 27 Jan 2019, at 06:57, airflowuser <
> airflowuser@protonmail.com
> > > >>>> .INVALID>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>> The Mesos executor is being used (at least by some
people) and
> > there
> > > >> is
> > > >>>>> an open PR to extend it:
> > > >>>>>> https://github.com/apache/airflow/pull/3739
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Sent with ProtonMail Secure Email.
> > > >>>>>>
> > > >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>>> What about Dask Executor as well.
> > > >>>>>>>
> > > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> > > >>>> daniel.imberman@gmail.com
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> @sid basically my response lol
> > > >>>>>>>> I haven't seen any emails from any users and
if it's been
> broken
> > > for
> > > >>>>> two
> > > >>>>>>>> versions then there doesn't seem to be enough
interest to
> > maintain
> > > >>>> it.
> > > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand
r39132@gmail.com
> > wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> LOL :_0
> > > >>>>>>>>> -s
> > > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke
de Bruin
> > bdbruin@gmail.com
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> I agree. Maybe we should make a choice
on what we consider
> 1st
> > > >>>> class
> > > >>>>>>>>>> executors and move others to contrib
(I'm looking at you
> > celery)
> > > >>>>>>>>>> Sent from my iPhone
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash
Berlin-Taylor ash@apache.org
> > > >> wrote:
> > > >>>>>>>>>>> Is anyone using the Mesos Executor?
I think we should
> > deprecate
> > > >>>> and
> > > >>>>>>>>>>> remove it.
> > > >>>>>>>>>>> The reason I say that is I have
a feeling it's been broken
> > > since
> > > >>>>>>>>>>> 1.9.0
> > > >>>>>>>>>
> > > >>>>>>>>>> and no-one has noticed:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> class MesosExecutor(BaseExecutor,
LoginMixin):
> > > >>>>>>>>>>> def start(self):
> > > >>>>>>>>>>> # ...
> > > >>>>>>>>>>> self.log.info(
> > > >>>>>>>>>>> 'MesosFramework master : %s, name
: %s, cpu : %s, mem :
> > > >>>>>>>>>>> %s,
> > > >>>>>>>>>
> > > >>>>>>>>>> checkpoint : %s',
> > > >>>>>>>>>>
> > > >>>>>>>>>>>         master, framework.name,
> > > >>>>>>>>>>>         str(task_cpu), str(task_memory),
> > > >>>> str(framework.checkpoint)
> > > >>>>>>>>>>>     )
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> That mixin should have been LoggingMixin,
not LoginMixin.
> But
> > > >>>> given
> > > >>>>>>>>>>> this
> > > >>>>>>>>>>> bug has been in since
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> > > >>>>>>>>
> > > >>>>>>>>>> which was in the 1.9.0 release I propose
we simply remove
> this
> > > >>>>> executor
> > > >>>>>>>>>> from master.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Thoughts?
> > > >>>>>>>>>>> Ash
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message