incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leo Neumeyer (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (S4-22) Adaptor
Date Wed, 02 Nov 2011 22:35:32 GMT
Adaptor
-------

                 Key: S4-22
                 URL: https://issues.apache.org/jira/browse/S4-22
             Project: Apache S4
          Issue Type: Improvement
    Affects Versions: 0.5
            Reporter: Leo Neumeyer
            Assignee: Bruce Robbins


Need an adaptor for v0.5

Idea I posted earlier:


What do you think of this idea for a simple adaptor:

- Adaptor extends App
- Adaptor can send events but not receive (for now)
- Adaptor is deployed as a regular App to the S4 cluster and as an
Adaptor type  in a host (separate from the S4 cluster).
- Adaptor, unlike regular apps, can accept event data (in any format)
directly, not via comm layer.
- Input data is transformed into S4 events using a modular approach
and by providing standard modules such as JSON.
- Output events are exposed using EventSource and consumed by other
apps without even knowing that they are Adaptors (only the App type is
exposed in the cluster).
- S4 events can be processed locally using PEs and Streams as usual.
(We kind of need to get a local Sender for the local PEs and a
standard cluster Sender for the EventSource object.)

So why this approach?

The GOOD:
- Seems to be the least disruptive way to inject external events
- Apps can easily consume the events in a modular way without any
dependencies. Getting events from an adaptor or from another app is
identical.
- The adaptor would be packaged and deployed to the cluster as if it
was an App (no incremental cost)
- The adaptor can do preprocessing using the same programming model
and can reuse PEs.

The CHALLENGE:
-  We need to also deploy the Adaptor in a separate host. On the other
hand, this is inevitable. At least we use the same approach instead of
creating a different system.
-  The Adaptor will need to be integrated with ZK to get the physical addresses.
-  We need to deal with two senders.

for later: two-way communication and adapter clusters.

thoughts?


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message