airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Date Wed, 27 Feb 2019 12:51:10 GMT
I say lets remove it in 2.0.

> On 27 Feb 2019, at 11:50, Driesprong, Fokko <fokko@driesprong.frl> wrote:
> 
> Are we going to remove the MesosExecutor? I think it is a safe bet since in
> the Survey of Ash (thanks again for that!), it also shows that nobody is
> really using it.
> 
> https://jira.apache.org/jira/browse/AIRFLOW-3969
> 
> Cheers, Fokko
> 
> Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <ash@apache.org>:
> 
>> I’m glad yarn wasn’t the only option - it would have meant I’d have never
>> been in a position to use Airflow! (Many of our workflows don’t touch
>> EMR/Hadoop, and running Celery is much more of a known element to a python
>> developer than configuring Yarn)
>> 
>>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <maximebeauchemin@gmail.com>
>> wrote:
>>> 
>>> From memory, I think MesosExecutor depends on pickling to get DAG
>>> definitions to workers, which we should also deprecate.
>>> 
>>> About CeleryExecutor, we never had the intention to make it the
>> recommended
>>> option for production early on. The intent back in 2014 was to write a
>>> YarnExecutor quickly (that was before the rise of k8s), but realized that
>>> container support wasn't ready in Yarn at the time, and had hard
>>> restrictions on what we could install on Hadoop/Yarn data nodes, so we
>>> moved forward with Celery.
>>> 
>>> I think it's practical to have LocalExecutor around though for unit test,
>>> local use, and for POCs.
>>> 
>>> Max
>>> 
>>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
>>> o.girardot@lateral-thoughts.com> wrote:
>>> 
>>>> Hi, we tried to use it for a new project 1year ago, but it was too
>> broken
>>>> to  be usable, and we didn't have the time to make it work.
>>>> Of course we are still hoping to use it one day, but that doesn't mean
>> it
>>>> should not move to contrib, it was already clear from the documentation
>>>> that it was not a first class citizen.
>>>> 
>>>> Regards,
>>>> 
>>>> Olivier.
>>>> 
>>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <ash@apache.org>
a
>> écrit
>>>> :
>>>> 
>>>>> That PR pre-dates 1.9.0 being released.
>>>>> 
>>>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
>>>> .INVALID>
>>>>> wrote:
>>>>>> 
>>>>>> The Mesos executor is being used (at least by some people) and there
>> is
>>>>> an open PR to extend it:
>>>>>> https://github.com/apache/airflow/pull/3739
>>>>>> 
>>>>>> 
>>>>>> Sent with ProtonMail Secure Email.
>>>>>> 
>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <kaxilnaik@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> What about Dask Executor as well.
>>>>>>> 
>>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
>>>> daniel.imberman@gmail.com
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> @sid basically my response lol
>>>>>>>> I haven't seen any emails from any users and if it's been
broken for
>>>>> two
>>>>>>>> versions then there doesn't seem to be enough interest to
maintain
>>>> it.
>>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com
wrote:
>>>>>>>>> 
>>>>>>>>> LOL :_0
>>>>>>>>> -s
>>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I agree. Maybe we should make a choice on what we
consider 1st
>>>> class
>>>>>>>>>> executors and move others to contrib (I'm looking
at you celery)
>>>>>>>>>> Sent from my iPhone
>>>>>>>>>> 
>>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
>> wrote:
>>>>>>>>>>> Is anyone using the Mesos Executor? I think we
should deprecate
>>>> and
>>>>>>>>>>> remove it.
>>>>>>>>>>> The reason I say that is I have a feeling it's
been broken since
>>>>>>>>>>> 1.9.0
>>>>>>>>> 
>>>>>>>>>> and no-one has noticed:
>>>>>>>>>> 
>>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
>>>>>>>>>>> def start(self):
>>>>>>>>>>> # ...
>>>>>>>>>>> self.log.info(
>>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu :
%s, mem :
>>>>>>>>>>> %s,
>>>>>>>>> 
>>>>>>>>>> checkpoint : %s',
>>>>>>>>>> 
>>>>>>>>>>>         master, framework.name,
>>>>>>>>>>>         str(task_cpu), str(task_memory),
>>>> str(framework.checkpoint)
>>>>>>>>>>>     )
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> That mixin should have been LoggingMixin, not
LoginMixin. But
>>>> given
>>>>>>>>>>> this
>>>>>>>>>>> bug has been in since
>>>>>>>> 
>>>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>>>>>> 
>>>>>>>>>> which was in the 1.9.0 release I propose we simply
remove this
>>>>> executor
>>>>>>>>>> from master.
>>>>>>>>>> 
>>>>>>>>>>> Thoughts?
>>>>>>>>>>> Ash
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 


Mime
View raw message