airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Date Wed, 27 Feb 2019 11:50:55 GMT
Are we going to remove the MesosExecutor? I think it is a safe bet since in
the Survey of Ash (thanks again for that!), it also shows that nobody is
really using it.

https://jira.apache.org/jira/browse/AIRFLOW-3969

Cheers, Fokko

Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <ash@apache.org>:

> I’m glad yarn wasn’t the only option - it would have meant I’d have never
> been in a position to use Airflow! (Many of our workflows don’t touch
> EMR/Hadoop, and running Celery is much more of a known element to a python
> developer than configuring Yarn)
>
> > On 11 Feb 2019, at 17:14, Maxime Beauchemin <maximebeauchemin@gmail.com>
> wrote:
> >
> > From memory, I think MesosExecutor depends on pickling to get DAG
> > definitions to workers, which we should also deprecate.
> >
> > About CeleryExecutor, we never had the intention to make it the
> recommended
> > option for production early on. The intent back in 2014 was to write a
> > YarnExecutor quickly (that was before the rise of k8s), but realized that
> > container support wasn't ready in Yarn at the time, and had hard
> > restrictions on what we could install on Hadoop/Yarn data nodes, so we
> > moved forward with Celery.
> >
> > I think it's practical to have LocalExecutor around though for unit test,
> > local use, and for POCs.
> >
> > Max
> >
> > On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> > o.girardot@lateral-thoughts.com> wrote:
> >
> >> Hi, we tried to use it for a new project 1year ago, but it was too
> broken
> >> to  be usable, and we didn't have the time to make it work.
> >> Of course we are still hoping to use it one day, but that doesn't mean
> it
> >> should not move to contrib, it was already clear from the documentation
> >> that it was not a first class citizen.
> >>
> >> Regards,
> >>
> >> Olivier.
> >>
> >> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <ash@apache.org> a
> écrit
> >> :
> >>
> >>> That PR pre-dates 1.9.0 being released.
> >>>
> >>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
> >> .INVALID>
> >>> wrote:
> >>>>
> >>>> The Mesos executor is being used (at least by some people) and there
> is
> >>> an open PR to extend it:
> >>>> https://github.com/apache/airflow/pull/3739
> >>>>
> >>>>
> >>>> Sent with ProtonMail Secure Email.
> >>>>
> >>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <kaxilnaik@gmail.com>
> >>> wrote:
> >>>>
> >>>>> What about Dask Executor as well.
> >>>>>
> >>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
> >> daniel.imberman@gmail.com
> >>>>> wrote:
> >>>>>
> >>>>>> @sid basically my response lol
> >>>>>> I haven't seen any emails from any users and if it's been broken
for
> >>> two
> >>>>>> versions then there doesn't seem to be enough interest to maintain
> >> it.
> >>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com
wrote:
> >>>>>>>
> >>>>>>> LOL :_0
> >>>>>>> -s
> >>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I agree. Maybe we should make a choice on what we consider
1st
> >> class
> >>>>>>>> executors and move others to contrib (I'm looking at
you celery)
> >>>>>>>> Sent from my iPhone
> >>>>>>>>
> >>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
> wrote:
> >>>>>>>>> Is anyone using the Mesos Executor? I think we should
deprecate
> >> and
> >>>>>>>>> remove it.
> >>>>>>>>> The reason I say that is I have a feeling it's been
broken since
> >>>>>>>>> 1.9.0
> >>>>>>>
> >>>>>>>> and no-one has noticed:
> >>>>>>>>
> >>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
> >>>>>>>>> def start(self):
> >>>>>>>>> # ...
> >>>>>>>>> self.log.info(
> >>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s,
mem :
> >>>>>>>>> %s,
> >>>>>>>
> >>>>>>>> checkpoint : %s',
> >>>>>>>>
> >>>>>>>>>          master, framework.name,
> >>>>>>>>>          str(task_cpu), str(task_memory),
> >> str(framework.checkpoint)
> >>>>>>>>>      )
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin.
But
> >> given
> >>>>>>>>> this
> >>>>>>>>> bug has been in since
> >>>>>>
> >>>>>>
> >>>
> >>
> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
> >>>>>>
> >>>>>>>> which was in the 1.9.0 release I propose we simply remove
this
> >>> executor
> >>>>>>>> from master.
> >>>>>>>>
> >>>>>>>>> Thoughts?
> >>>>>>>>> Ash
> >>>>
> >>>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message