falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajay Yadav <ajayn...@gmail.com>
Subject Re: [DISCUSS] Orchestration in Falcon
Date Tue, 07 Apr 2015 06:12:00 GMT
+1



On Tue, Apr 7, 2015 at 11:27 AM, Pallavi Rao <pallavi.rao@inmobi.com> wrote:

> Hi,
> I was recently looking at some of the use cases at InMobi and how to
> enhance Falcon to accommodate those and I realized that due to our
> dependency on Oozie coordinator, some of these cannot be easily achieved or
> take a much longer cycle as we have to wait for Oozie to add some
> functionality.
>
> I was pointed to this thread that dates slightly before my time in Falcon (
> https://www.mail-archive.com/dev@falcon.incubator.apache.org/msg09268.html
> ).
> I wanted to reopen the thread for discussion, with my 2 cents:
>
>    1. Some of the scheduling primitives that are already mentioned in the
>    thread, especially, support for a-periodic datasets or external
> triggering
>    mechanisms are not available in Oozie. It might not even be a natural
> fit
>    for Oozie to add these.
>    2. Adding new primitives in Falcon becomes harder and longer as we
>    completely depend on Oozie for the same. Extensibility of Falcon is
> stunted.
>    3. Oozie has very limited support for throttling resource utilization.
>    We can only control the no. of parallel instances of a coordinator job.
>    4. Oozie currently has no notion of inter dependency of
>    instances/workflows, whereas, in Falcon, it will be very useful to
>    gate/throttle based on the interdependency. For example, re-run a
> pipeline
>    (or a subset) or throttle resource utilization of a pipeline when in
>    "backlog catchup" mode.
>    5. We end up with bugs like FALCON-1127
>    <https://issues.apache.org/jira/browse/FALCON-1127>, because Falcon
>    constantly needs to play catchup with Oozie changes.
>
> On the thread, most people did seem to be in favor of a native scheduler in
> Falcon. If you all think this is useful, I'll volunteer to start work on
> this and we can build out a scheduler/orchestrator in Falcon that can open
> up a whole lot of possibilities for Falcon users.
>
> Thanks,
> Pallavi
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message