Return-Path: 
 <dev-return-7393-apmail-ignite-dev-archive=ignite.apache.org@ignite.apache.org>
X-Original-To: apmail-ignite-dev-archive@minotaur.apache.org
Delivered-To: apmail-ignite-dev-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4ADBF18412
	for <apmail-ignite-dev-archive@minotaur.apache.org>;
 Wed, 23 Sep 2015 12:14:34 +0000 (UTC)
Received: (qmail 31279 invoked by uid 500); 23 Sep 2015 12:14:34 -0000
Delivered-To: apmail-ignite-dev-archive@ignite.apache.org
Received: (qmail 31240 invoked by uid 500); 23 Sep 2015 12:14:34 -0000
Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@ignite.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@ignite.apache.org>
List-Post: <mailto:dev@ignite.apache.org>
List-Id: <dev.ignite.apache.org>
Reply-To: dev@ignite.apache.org
Delivered-To: mailing list dev@ignite.apache.org
Received: (qmail 31228 invoked by uid 99); 23 Sep 2015 12:14:34 -0000
Received: from mail-relay.apache.org (HELO mail-relay.apache.org)
 (140.211.11.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Sep 2015 12:14:34 +0000
Received: from mail-la0-f51.google.com (mail-la0-f51.google.com
 [209.85.215.51])
	by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with
 ESMTPSA id 655791A025E
	for <dev@ignite.apache.org>; Wed, 23 Sep 2015 12:14:33 +0000 (UTC)
Received: by lagj9 with SMTP id j9so47856743lag.2
        for <dev@ignite.apache.org>; Wed, 23 Sep 2015 05:14:31 -0700 (PDT)
X-Gm-Message-State: 
 ALoCoQmCnU9XFzneJ8ALdbGgc5tl8bLjtNEeijAwVT03qWfgjc7JXvmjy4eEY/kfcWOAhQsucEfl
X-Received: by 10.152.5.170 with SMTP id t10mr11175350lat.112.1443010471973;
 Wed, 23 Sep 2015 05:14:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.112.74.135 with HTTP; Wed, 23 Sep 2015 05:14:12 -0700 (PDT)
X-Originating-IP: [85.155.76.117]
In-Reply-To: 
 <CA+0=VoUq+G8FQZEYU-o0JpyviMBajAFKj4oWPwmRCJ7GvdeVPQ@mail.gmail.com>
References: 
 <CAKoNANs+r1ctgeqFof7dCSBq5JF14XSmFb6XXJ1C7ZtKKL11Tw@mail.gmail.com>
 <CABthXtDqrDgs31tJT-f4zn6a-3hT11=fuQhZ5YrNeVgMhMfbFw@mail.gmail.com>
 <AE27CEFE-213C-40EC-918D-D6968F8A5443@gmail.com>
 <CAKoNANtVhX+YAp8jxrsjG+ia7uz=Q=AAN0KSRkzj+jJUZROuuQ@mail.gmail.com>
 <CA+eZwrHXZ4o1e71b8uCA4PP9quARYR_djz8G=2qV+2=n70-RLQ@mail.gmail.com>
 <CAAgW0dZJ16Wu1UqLBA3TsTkM-vq-LDiZeW0+vpirfnVTFKB66w@mail.gmail.com>
 <CA+0=VoUq+G8FQZEYU-o0JpyviMBajAFKj4oWPwmRCJ7GvdeVPQ@mail.gmail.com>
From: Raul Kripalani <raulk@apache.org>
Date: Wed, 23 Sep 2015 13:14:12 +0100
Message-ID: 
 <CAKoNANunCkumJA6u7H5NgOQFyE+2M2s=wNhL7T6Mhwr7UP+fMQ@mail.gmail.com>
Subject: Re: Brainstorming about Reactive Streams
To: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary=089e013d18b096f2030520690e33

--089e013d18b096f2030520690e33
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

First things first, let me clarify that I didn't request the feature myself
so I'm not advocating it ;-)

We have a vague proposition in a JIRA ticket, and my intention is to try
give it some objective essence. The team can happily close the ticket with
a "Won't Fix" if we feel that there's no compelling benefit.

That said, the more I think about it, the more I believe Reactive Streams
and Ignite are a valuable combination.

*#1 Compute Jobs*

Reactive Streams would give compute jobs the ability to return partial
results as they are being generated. Basically an API like:

    Observable<String> ob =3D grid.compute().submit(Abc.class,
"foo").toObservable();

would return an Observable to which we can then subscribe to get results as
they are being generated:

    ob.subscribe((result) -> System.out.println("Got a result " + result));

The client would get results as they are being emitted by the compute job
server-side.

Of course this is the simplest example. Once we have an Observable we can
do lots and lots of streaming operations on it: merge, drop, folds, etc.
and even combine those operations with other compute jobs. Full set of
operators supported by RxJava:
http://reactivex.io/documentation/operators.html.

So basically a client could send 2, 3, 4... compute jobs and combine their
results together in a streaming fashion:

    Observable<String> ob1 =3D grid.compute().submit(Abc.class,
"foo").toObservable();
    Observable<String> ob2 =3D grid.compute().submit(Def.class,
"foo").toObservable();
    Observable<String> ob3 =3D grid.compute().submit(Ghi.class,
"foo").toObservable();

    ob1.flatMap((result) =3D> Observable.just("Observable 1 emitted: " +
result))
          .merge(ob2, ob3.skipLast(2))
          .subscribe((result) -> System.out.println("Got a result " +
result));

This example would submit 3 jobs, and would combine the streams of all 3
into a single one, but applying a transformation to ob1 and skipping the
last 2 items emitted from ob3.

*#2 Data Streamers*

Similar to above, but for data streamers. Currently our streaming logic is
simple: get a message, transform it and add it to cache. Some streamers
allow ignoring messages, but that's all.

But maybe a user doesn't want to store every single incoming item in the
cache. Maybe they want to store a sliding window result, i.e. receive for 5
seconds, calculate moving averages and store that in the cache.

Or maybe they want to combine multiple data streams together, e.g. combine
3 types of MQTT messages within a window of 10 seconds and only store the
result of an aggregation.

---

I also mentioned messaging and events in my initial email. The rationale
would be similar to the above.

P.S.: For my crude API examples, I'm using RxJava =E2=80=93 as that's what =
I'm
familiar with =E2=80=93 not the Reactive Streams API.
P.S. 2: We should put all ideas on a Wiki page =E2=80=93 even if we decide =
to
discard the implementation for now.

Regards,

*Ra=C3=BAl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk

On Mon, Sep 21, 2015 at 4:49 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> From my standpoint, I would like to see some API examples. How do you
> envision reactive stream support in Ignite at the API level?
>
> D.
>
> On Sun, Sep 20, 2015 at 9:59 PM, Lalit Kumar Jha <lalitj.dev@gmail.com>
> wrote:
>
> > Yes, lets first explore how reactive-streams can be useful for Ignite.
> >
> > Adding one more candidate to list http://projectreactor.io/docs/, as
> this
> > project:
> > Is for fast-data applications
> > Team are core contributors to the Reactive Streams Specification
> > <https://github.com/reactive-streams/reactive-streams>
> >
> >
> >
> > On Sun, Sep 20, 2015 at 10:33 PM, Sergi Vladykin <
> sergi.vladykin@gmail.com
> > >
> > wrote:
> >
> > > Guys,
> > >
> > > I think you are digging too deep in implementation possibilities righ=
t
> > > away, may be lets start from simpler things?
> > > I don't know much about reactive streams and probably not everyone in
> the
> > > community that educated as well :)
> > >
> > > It would be nice to start with a quick overview of what are they and
> why
> > in
> > > makes sense to
> > > to add such a functionality to Ignite? What real use cases do you see=
?
> > How
> > > can Ignite
> > > be useful for reactive streams users and vice versa?
> > >
> > > Lets get clear answers to these questions first to avoid writing
> useless
> > > code.
> > >
> > > Sergi
> > >
> > > 2015-09-20 18:55 GMT+03:00 Raul Kripalani <raul@evosent.com>:
> > >
> > > > Yeah, the idea would be to wrap RxJava 1.0 with the adapter and onl=
y
> > use
> > > > the Reactive Streams API where possible from within Ignite.
> > > >
> > > > Nevertheless, the ReactiveX API would also be accessible in case we
> > want
> > > to
> > > > do stuff with Hystrix too.
> > > >
> > > > Ra=C3=BAl.
> > > > On 20 Sep 2015 16:39, "Vishal Garg" <gargvish@gmail.com> wrote:
> > > >
> > > > > https://github.com/ReactiveX/RxJava/wiki/R
> > > > > eactive-Streams
> > > > > +1 on your approach ( if we are wrapping 1.0 with adapters??)
> > > > > Raul it looks like there is a path to convergence in RxJava 2.0,
> but
> > > > above
> > > > > link mentions that you could use the standard + integration modul=
es
> > > with
> > > > > wrappers on core 1.0 Apis. Where the modules could be Akka or
> RxJava
> > > > > Vishal
> > > > >
> > > > >
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Sep 20, 2015, at 7:03 AM, Gianfranco Murador <
> > > > > murador.gianfranco@gmail.com> wrote:
> > > > > >
> > > > > > Hy Raul, it could be a valid extension of the Service Grid of
> > Ignite
> > > (
> > > > > if I
> > > > > > understand the point), imho.
> > > > > >
> > > > > > 2015-09-20 14:37 GMT+02:00 Raul Kripalani <raulk@apache.org>:
> > > > > >
> > > > > >> Hey Igniters,
> > > > > >>
> > > > > >> Lalit Kumar Jha expressed interest [1] in contributing to tick=
et
> > > > > IGNITE-815
> > > > > >> [2] which aims to bring in Reactive Streams to Ignite.
> > > > > >>
> > > > > >> I wanted to start a brainstorm about which functionalities we'=
d
> > like
> > > > to
> > > > > >> cover.
> > > > > >>
> > > > > >> Off the top of my head, I can think of merging, filtering,
> > sorting,
> > > > > >> batching, grouping, etc. on Data Streams, Ignite Messages and
> > > Events.
> > > > > Also,
> > > > > >> integration with Continuous Queries would be useful.
> > > > > >>
> > > > > >> Furthermore, I wanted to ask the community if at this point we
> > feel
> > > > > >> comfortable with Reactive Streams per-se or we prefer to go wi=
th
> > > > RxJava
> > > > > for
> > > > > >> now. Reactive Streams is a proposed standard whose main Java
> > > > > implementation
> > > > > >> and is slowly gaining traction (e.g. vert.x [3] and MongoDB
> > > > integrations
> > > > > >> [4]), whereas RxJava is an already mature and production-teste=
d
> > > > > >> implementation of the initial ReactiveX design by Microsoft [5=
]
> > with
> > > > > 7000+
> > > > > >> stars on GitHub.
> > > > > >>
> > > > > >> RxJava is the foundation of Hystrix [6], a powerful library fo=
r
> > > > latency
> > > > > and
> > > > > >> fault tolerance management. RxJava does not implement Reactive
> > > Streams
> > > > > as
> > > > > >> of 1.x (as they predated it), but they've vowed support for 2.=
x
> > [7].
> > > > > >> Moreover, there's already an adapter which passes the TCK [8].
> > > > > >>
> > > > > >> My take is as follows. I'd like to follow the Reactive Streams
> > > > standard,
> > > > > >> but I wouldn't like to lose support for Hystrix (hopefully it'=
ll
> > > > > eventually
> > > > > >> be migrated to Reactive Streams too). So an option is to go wi=
th
> > the
> > > > > RxJava
> > > > > >> + adapter design for now to get the best of both worlds. Once
> > RxJava
> > > > > 2.0 is
> > > > > >> out, we can drop the adapter out of the picture.
> > > > > >>
> > > > > >> Thoughts?
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Hello-Apache-Ignite=
-Dev-tp3137p3315.html
> > > > > >> [2] https://issues.apache.org/jira/browse/IGNITE-815
> > > > > >> [3] http://vertx.io/docs/vertx-reactive-streams/java/
> > > > > >> [4]
> https://mongodb.github.io/mongo-java-driver-reactivestreams/
> > > > > >> [5] https://msdn.microsoft.com/en-us/data/gg577609.aspx
> > > > > >> [6] https://github.com/Netflix/Hystrix
> > > > > >> [7] https://github.com/ReactiveX/RxJava/wiki/Reactive-Streams
> > > > > >> [8] https://github.com/ReactiveX/RxJavaReactiveStreams
> > > > > >>
> > > > > >> Regards,
> > > > > >>
> > > > > >> *Ra=C3=BAl Kripalani*
> > > > > >> Apache Camel PMC Member & Committer | Enterprise Architect, Op=
en
> > > > Source
> > > > > >> Integration specialist
> > > > > >> http://about.me/raulkripalani |
> > > > > http://www.linkedin.com/in/raulkripalani
> > > > > >> http://blog.raulkr.net | twitter: @raulvk
> > > > > >>
> > > > >
> > > >
> > >
> >
>

--089e013d18b096f2030520690e33--