polygene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niclas Hedhman <nic...@hedhman.org>
Subject Re: SPI for EventDriven apps
Date Fri, 30 Mar 2018 01:39:11 GMT
Ok, some initial findings about running Polygene on Flink;

This is not going to be easy. Flink by default requires the "operators"
(functions/transformations, the blocks in the stream processing chain) to
be Serializable with java.io.ObjectOutputStream. That is not going to
happen any time soon with Polygene. The alternative (I think ) is to create
a custom TaskManager, possibly even a custom JobManager, which seems like a
major undertaking, with Scala actors (Akka?) and heavy use of
java.io.Serializable. At this point in time, I am not even sure that this
is at all possible even at that level of integration.

So, I am not going to bother at this time.

That said, having a Kafka integration, as described in this thread is
something that I will attempt.


Cheers
Niclas

On Wed, Mar 28, 2018 at 6:29 PM, Niclas Hedhman <niclas@hedhman.org> wrote:

> Gang,
>
> I have been toying with trying to run Polygene on a Flink cluster (no
> success yet), and part of that is coming back to having a pluggable
> Messaging/Event SPI, which is pluggable in the similar fashion to
> persistence.
>
> I am also getting a clearer point what is needed, and why we need a SPI
> and not just one or more libraries for different messaging systems.
>
> My initial (and possibly naive) thoughts are as follows;
>
> It all comes down to UnitOfWork and the fact that Queue readers are in
> their own threads.
>
> 1. A Consumer impl starts a UoW, starts a transaction on the queue*
> 2. Polls the Queue (potentially blocking)
> 3. Data read is deserialized**
> 4. ValueComposite is forwarded to all listeners***
> 5. Any kind of regular Polygene happens
> 6. Persistence might be invoked and part of the UoW
> 7. Producer impl might be called to forward a message to another
> queue/topic
> 8. Eventually returns to Consumer and the UoW.complete() happens, which
> needs to handle the transaction commit to all individual parts, of
> UoW.cancel() to rollback all parts.
>
>
> * Does it need to start transaction on any destination queues as well?
>
> ** Several issue here;
>     a. Visibility from where?
>     b. Multiple types on same queue, how to resolve?
>     c. Data formats not compatible with ValueComposites, such as raw
> binary.
>
> *** addListener or to all Services with some tagging?
>
>
> Non-trivial I would think, but not excessive by Polygene standards. :-)
>
> WDYAT?
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>



-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message