falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: [DISCUSS] Orchestration in Falcon
Date Tue, 07 Apr 2015 06:50:34 GMT
+1

I would like to help there ! ;)

Regards
JB

On 04/07/2015 08:47 AM, Zulfiqar Ali wrote:
> +1
>
> On Tue, Apr 7, 2015 at 11:42 AM, Ajay Yadav <ajaynsit@gmail.com> wrote:
>
>> +1
>>
>>
>>
>> On Tue, Apr 7, 2015 at 11:27 AM, Pallavi Rao <pallavi.rao@inmobi.com>
>> wrote:
>>
>>> Hi,
>>> I was recently looking at some of the use cases at InMobi and how to
>>> enhance Falcon to accommodate those and I realized that due to our
>>> dependency on Oozie coordinator, some of these cannot be easily achieved
>> or
>>> take a much longer cycle as we have to wait for Oozie to add some
>>> functionality.
>>>
>>> I was pointed to this thread that dates slightly before my time in
>> Falcon (
>>>
>> https://www.mail-archive.com/dev@falcon.incubator.apache.org/msg09268.html
>>> ).
>>> I wanted to reopen the thread for discussion, with my 2 cents:
>>>
>>>     1. Some of the scheduling primitives that are already mentioned in the
>>>     thread, especially, support for a-periodic datasets or external
>>> triggering
>>>     mechanisms are not available in Oozie. It might not even be a natural
>>> fit
>>>     for Oozie to add these.
>>>     2. Adding new primitives in Falcon becomes harder and longer as we
>>>     completely depend on Oozie for the same. Extensibility of Falcon is
>>> stunted.
>>>     3. Oozie has very limited support for throttling resource utilization.
>>>     We can only control the no. of parallel instances of a coordinator
>> job.
>>>     4. Oozie currently has no notion of inter dependency of
>>>     instances/workflows, whereas, in Falcon, it will be very useful to
>>>     gate/throttle based on the interdependency. For example, re-run a
>>> pipeline
>>>     (or a subset) or throttle resource utilization of a pipeline when in
>>>     "backlog catchup" mode.
>>>     5. We end up with bugs like FALCON-1127
>>>     <https://issues.apache.org/jira/browse/FALCON-1127>, because Falcon
>>>     constantly needs to play catchup with Oozie changes.
>>>
>>> On the thread, most people did seem to be in favor of a native scheduler
>> in
>>> Falcon. If you all think this is useful, I'll volunteer to start work on
>>> this and we can build out a scheduler/orchestrator in Falcon that can
>> open
>>> up a whole lot of possibilities for Falcon users.
>>>
>>> Thanks,
>>> Pallavi
>>>
>>> --
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>> the
>>> use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally
>> privileged
>>> information. If you are not the intended recipient you are hereby
>> notified
>>> that any disclosure, copying, distribution or taking any action in
>> reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete
>> transmission
>>> of the information contained in this communication nor for any delay in
>> its
>>> receipt.
>>>
>>
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message