airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Uellendall <felix.uellend...@gmx.de>
Subject Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?
Date Sat, 13 Apr 2019 09:50:15 GMT
+1 (non-binding)

Am 12/04/2019 um 23:42 schrieb Tao Feng:
> +1 on removing mesos executor.
>
> On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman <daniel.imberman@gmail.com>
> wrote:
>
>> Since we are now doing version 2.0 spring cleaning can we revisit this?
>>
>> On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> +1 as it relies on pickles
>>>
>>> On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <ash@apache.org>
>> wrote:
>>>> I say lets remove it in 2.0.
>>>>
>>>>> On 27 Feb 2019, at 11:50, Driesprong, Fokko <fokko@driesprong.frl>
>>>> wrote:
>>>>> Are we going to remove the MesosExecutor? I think it is a safe bet
>>> since
>>>> in
>>>>> the Survey of Ash (thanks again for that!), it also shows that nobody
>>> is
>>>>> really using it.
>>>>>
>>>>> https://jira.apache.org/jira/browse/AIRFLOW-3969
>>>>>
>>>>> Cheers, Fokko
>>>>>
>>>>> Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <
>> ash@apache.org
>>>> :
>>>>>> I’m glad yarn wasn’t the only option - it would have meant I’d
have
>>>> never
>>>>>> been in a position to use Airflow! (Many of our workflows don’t
>> touch
>>>>>> EMR/Hadoop, and running Celery is much more of a known element to
a
>>>> python
>>>>>> developer than configuring Yarn)
>>>>>>
>>>>>>> On 11 Feb 2019, at 17:14, Maxime Beauchemin <
>>>> maximebeauchemin@gmail.com>
>>>>>> wrote:
>>>>>>>  From memory, I think MesosExecutor depends on pickling to get
DAG
>>>>>>> definitions to workers, which we should also deprecate.
>>>>>>>
>>>>>>> About CeleryExecutor, we never had the intention to make it the
>>>>>> recommended
>>>>>>> option for production early on. The intent back in 2014 was to
>> write
>>> a
>>>>>>> YarnExecutor quickly (that was before the rise of k8s), but
>> realized
>>>> that
>>>>>>> container support wasn't ready in Yarn at the time, and had hard
>>>>>>> restrictions on what we could install on Hadoop/Yarn data nodes,
so
>>> we
>>>>>>> moved forward with Celery.
>>>>>>>
>>>>>>> I think it's practical to have LocalExecutor around though for
unit
>>>> test,
>>>>>>> local use, and for POCs.
>>>>>>>
>>>>>>> Max
>>>>>>>
>>>>>>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot <
>>>>>>> o.girardot@lateral-thoughts.com> wrote:
>>>>>>>
>>>>>>>> Hi, we tried to use it for a new project 1year ago, but it
was too
>>>>>> broken
>>>>>>>> to  be usable, and we didn't have the time to make it work.
>>>>>>>> Of course we are still hoping to use it one day, but that
doesn't
>>> mean
>>>>>> it
>>>>>>>> should not move to contrib, it was already clear from the
>>>> documentation
>>>>>>>> that it was not a first class citizen.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Olivier.
>>>>>>>>
>>>>>>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <ash@apache.org>
>> a
>>>>>> écrit
>>>>>>>> :
>>>>>>>>
>>>>>>>>> That PR pre-dates 1.9.0 being released.
>>>>>>>>>
>>>>>>>>>> On 27 Jan 2019, at 06:57, airflowuser <
>> airflowuser@protonmail.com
>>>>>>>> .INVALID>
>>>>>>>>> wrote:
>>>>>>>>>> The Mesos executor is being used (at least by some
people) and
>>> there
>>>>>> is
>>>>>>>>> an open PR to extend it:
>>>>>>>>>> https://github.com/apache/airflow/pull/3739
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>>>>>
>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <
>>>> kaxilnaik@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>> What about Dask Executor as well.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman <
>>>>>>>> daniel.imberman@gmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> @sid basically my response lol
>>>>>>>>>>>> I haven't seen any emails from any users
and if it's been
>> broken
>>>> for
>>>>>>>>> two
>>>>>>>>>>>> versions then there doesn't seem to be enough
interest to
>>> maintain
>>>>>>>> it.
>>>>>>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand
r39132@gmail.com
>>> wrote:
>>>>>>>>>>>>> LOL :_0
>>>>>>>>>>>>> -s
>>>>>>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke
de Bruin
>>> bdbruin@gmail.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree. Maybe we should make a choice
on what we consider
>> 1st
>>>>>>>> class
>>>>>>>>>>>>>> executors and move others to contrib
(I'm looking at you
>>> celery)
>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash
Berlin-Taylor ash@apache.org
>>>>>> wrote:
>>>>>>>>>>>>>>> Is anyone using the Mesos Executor?
I think we should
>>> deprecate
>>>>>>>> and
>>>>>>>>>>>>>>> remove it.
>>>>>>>>>>>>>>> The reason I say that is I have
a feeling it's been broken
>>>> since
>>>>>>>>>>>>>>> 1.9.0
>>>>>>>>>>>>>> and no-one has noticed:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> class MesosExecutor(BaseExecutor,
LoginMixin):
>>>>>>>>>>>>>>> def start(self):
>>>>>>>>>>>>>>> # ...
>>>>>>>>>>>>>>> self.log.info(
>>>>>>>>>>>>>>> 'MesosFramework master : %s,
name : %s, cpu : %s, mem :
>>>>>>>>>>>>>>> %s,
>>>>>>>>>>>>>> checkpoint : %s',
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>          master, framework.name,
>>>>>>>>>>>>>>>          str(task_cpu), str(task_memory),
>>>>>>>> str(framework.checkpoint)
>>>>>>>>>>>>>>>      )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That mixin should have been LoggingMixin,
not LoginMixin.
>> But
>>>>>>>> given
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> bug has been in since
>>>>>>>>>>>>
>> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29
>>>>>>>>>>>>>> which was in the 1.9.0 release I
propose we simply remove
>> this
>>>>>>>>> executor
>>>>>>>>>>>>>> from master.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>> Ash
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>

Mime
View raw message