airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Date Tue, 12 Feb 2019 09:19:38 GMT
I’m glad yarn wasn’t the only option - it would have meant I’d have never been in a position
to use Airflow! (Many of our workflows don’t touch EMR/Hadoop, and running Celery is much
more of a known element to a python developer than configuring Yarn)

> On 11 Feb 2019, at 17:14, Maxime Beauchemin <maximebeauchemin@gmail.com> wrote:
> 
> From memory, I think MesosExecutor depends on pickling to get DAG
> definitions to workers, which we should also deprecate.
> 
> About CeleryExecutor, we never had the intention to make it the recommended
> option for production early on. The intent back in 2014 was to write a
> YarnExecutor quickly (that was before the rise of k8s), but realized that
> container support wasn't ready in Yarn at the time, and had hard
> restrictions on what we could install on Hadoop/Yarn data nodes, so we
> moved forward with Celery.
> 
> I think it's practical to have LocalExecutor around though for unit test,
> local use, and for POCs.
> 
> Max
> 
> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
> o.girardot@lateral-thoughts.com> wrote:
> 
>> Hi, we tried to use it for a new project 1year ago, but it was too broken
>> to  be usable, and we didn't have the time to make it work.
>> Of course we are still hoping to use it one day, but that doesn't mean it
>> should not move to contrib, it was already clear from the documentation
>> that it was not a first class citizen.
>> 
>> Regards,
>> 
>> Olivier.
>> 
>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <ash@apache.org> a écrit
>> :
>> 
>>> That PR pre-dates 1.9.0 being released.
>>> 
>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowuser@protonmail.com
>> .INVALID>
>>> wrote:
>>>> 
>>>> The Mesos executor is being used (at least by some people) and there is
>>> an open PR to extend it:
>>>> https://github.com/apache/airflow/pull/3739
>>>> 
>>>> 
>>>> Sent with ProtonMail Secure Email.
>>>> 
>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <kaxilnaik@gmail.com>
>>> wrote:
>>>> 
>>>>> What about Dask Executor as well.
>>>>> 
>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
>> daniel.imberman@gmail.com
>>>>> wrote:
>>>>> 
>>>>>> @sid basically my response lol
>>>>>> I haven't seen any emails from any users and if it's been broken
for
>>> two
>>>>>> versions then there doesn't seem to be enough interest to maintain
>> it.
>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39132@gmail.com wrote:
>>>>>>> 
>>>>>>> LOL :_0
>>>>>>> -s
>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbruin@gmail.com
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I agree. Maybe we should make a choice on what we consider
1st
>> class
>>>>>>>> executors and move others to contrib (I'm looking at you
celery)
>>>>>>>> Sent from my iPhone
>>>>>>>> 
>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor ash@apache.org
wrote:
>>>>>>>>> Is anyone using the Mesos Executor? I think we should
deprecate
>> and
>>>>>>>>> remove it.
>>>>>>>>> The reason I say that is I have a feeling it's been broken
since
>>>>>>>>> 1.9.0
>>>>>>> 
>>>>>>>> and no-one has noticed:
>>>>>>>> 
>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin):
>>>>>>>>> def start(self):
>>>>>>>>> # ...
>>>>>>>>> self.log.info(
>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem
:
>>>>>>>>> %s,
>>>>>>> 
>>>>>>>> checkpoint : %s',
>>>>>>>> 
>>>>>>>>>          master, framework.name,
>>>>>>>>>          str(task_cpu), str(task_memory),
>> str(framework.checkpoint)
>>>>>>>>>      )
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin.
But
>> given
>>>>>>>>> this
>>>>>>>>> bug has been in since
>>>>>> 
>>>>>> 
>>> 
>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>>>> 
>>>>>>>> which was in the 1.9.0 release I propose we simply remove
this
>>> executor
>>>>>>>> from master.
>>>>>>>> 
>>>>>>>>> Thoughts?
>>>>>>>>> Ash
>>>> 
>>>> 
>>> 
>>> 
>> 


Mime
View raw message