incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Srivastava <mo...@s4.io>
Subject Re: simple adaptor idea
Date Thu, 20 Oct 2011 01:18:53 GMT
It is a great idea. It can accumulate external events and the reuse of PE
will be helpful.

--
Mohit

On Thu, Oct 20, 2011 at 6:25 AM, Leo Neumeyer <leoneumeyer@gmail.com> wrote:

> What do you think of this idea for a simple adaptor:
>
> - Adaptor extends App
> - Adaptor can send events but not receive (for now)
> - Adaptor is deployed as a regular App to the S4 cluster and as an
> Adaptor type  in a host (separate from the S4 cluster).
> - Adaptor, unlike regular apps, can accept event data (in any format)
> directly, not via comm layer.
> - Input data is transformed into S4 events using a modular approach
> and by providing standard modules such as JSON.
> - Output events are exposed using EventSource and consumed by other
> apps without even knowing that they are Adaptors (only the App type is
> exposed in the cluster).
> - S4 events can be processed locally using PEs and Streams as usual.
> (We kind of need to get a local Sender for the local PEs and a
> standard cluster Sender for the EventSource object.)
>
> So why this approach?
>
> The GOOD:
> - Seems to be the least disruptive way to inject external events
> - Apps can easily consume the events in a modular way without any
> dependencies. Getting events from an adaptor or from another app is
> identical.
> - The adaptor would be packaged and deployed to the cluster as if it
> was an App (no incremental cost)
> - The adaptor can do preprocessing using the same programming model
> and can reuse PEs.
>
> The CHALLENGE:
> -  We need to also deploy the Adaptor in a separate host. On the other
> hand, this is inevitable. At least we use the same approach instead of
> creating a different system.
> -  The Adaptor will need to be integrated with ZK to get the physical
> addresses.
> -  We need to deal with two senders.
>
> for later: two-way communication and adapter clusters.
>
> thoughts?
>
> --
>
> -leo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message