ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Read this if you want to integrate Ignite with other platforms (Python, R, etc.)
Date Wed, 25 Jan 2017 08:09:30 GMT

I see growing community interest to implementation of so-called "clients"
to other platforms, such as R, Python, etc.. Let me briefly describe
architecture of our existing integrations - C++ and .NET.

*1) Not a client!*
We do not have clients in classical sense. Our C++ and .NET integration are
fully-fledged platforms. C++/.NET node can act both as a client and as a

*2) JVM inside*
We didn't implement existing integrations form scratch. Instead we wrapped
Java into relevant platform's API. So when your .NET/C++ node is started,
it starts JVM in the same process and delegate most of the work to it.
We considering creation of another mode, something similar to thin client,
when platform will not have JVM inside, but instead will connect to
JVM-based node 1]. However, no design work have been started on it so far.

*3) Forward and reverse calls*
Ignite is complex product. We have methods with "forward" semantics, when
platform invoke JVM, e.g. Cache.get(). But we also have plenty of "reverse"
calls, when JVM need to perform JNI call to the platform, e.g. continuous
query notification.

*4) Binary everywhere*
We have full interoperability between platforms. Object created in .NET can
be accessed from Java and C++ and vice versa. To achieve this we
implemented special "binary protocol" on all platforms, so that every
platform serialize objects in the same format. If you are to implement
fully-fledged platform integration, you will have to implement binary
marshaller on your own.

*5) API layering*
We have two C++ API layers - "common" [2] and everything else [3]. Common
API is shared between C++ and .NET integrations. It is low-level interface
to communicate with JVM (i.e. to perform forward and reverse JNI calls). It
operates on pointers and JNI handles, and expects that you already
serialized your objects in binary format amd put it to some memory chunk
with certain layout. "Everything else" is high-level C++ API exposed to

*6) Do not expect SWIG to help you a lot!*
Taking all previous paragraphs in count - it is impossible to create
valuable working implementation with SWIG. You will have to do a lot more
than simple codegen.

*7) Any other ways to integrate?*
Yes, you can rely on our REST API [4]. We were planning to integrate with
Node.JS this way - you just send HTTP requests to Ignite node(s). Every
request is mapped to certain operation. However, this approach has two
serious limitations:
- It will be hard to employ advantages of binary interoperability
- It will be very hard to implement server -> client callbacks which are of
great importance for Ignite.
As such you will have problems implementing compute API, continuous queries
and so forth.

Please let me know if you have any further questions.


[1] https://issues.apache.org/jira/browse/IGNITE-3568
[3] https://github.com/apache/ignite/tree/master/modules/platforms/cpp
[4] https://apacheignite.readme.io/docs/rest-api

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message