ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shashank Gandham <shashankgand...@gmail.com>
Subject Re: Read this if you want to integrate Ignite with other platforms (Python, R, etc.)
Date Wed, 25 Jan 2017 09:12:10 GMT
So if we have to make full-fledged python platform for ignite. We can use C++ common  API to
integrate and have an high level API of Python. Object serialisation can be achieved in python
using libraries like marshal, pickle, shelve or something similar. Theoretically speaking
this should be possible, or am I missing something?

> On 25-Jan-2017, at 1:39 PM, Vladimir Ozerov <vozerov@gridgain.com> wrote:
> Igniters,
> I see growing community interest to implementation of so-called "clients"
> to other platforms, such as R, Python, etc.. Let me briefly describe
> architecture of our existing integrations - C++ and .NET.
> *1) Not a client!*
> We do not have clients in classical sense. Our C++ and .NET integration are
> fully-fledged platforms. C++/.NET node can act both as a client and as a
> server.
> *2) JVM inside*
> We didn't implement existing integrations form scratch. Instead we wrapped
> Java into relevant platform's API. So when your .NET/C++ node is started,
> it starts JVM in the same process and delegate most of the work to it.
> We considering creation of another mode, something similar to thin client,
> when platform will not have JVM inside, but instead will connect to
> JVM-based node 1]. However, no design work have been started on it so far.
> *3) Forward and reverse calls*
> Ignite is complex product. We have methods with "forward" semantics, when
> platform invoke JVM, e.g. Cache.get(). But we also have plenty of "reverse"
> calls, when JVM need to perform JNI call to the platform, e.g. continuous
> query notification.
> *4) Binary everywhere*
> We have full interoperability between platforms. Object created in .NET can
> be accessed from Java and C++ and vice versa. To achieve this we
> implemented special "binary protocol" on all platforms, so that every
> platform serialize objects in the same format. If you are to implement
> fully-fledged platform integration, you will have to implement binary
> marshaller on your own.
> *5) API layering*
> We have two C++ API layers - "common" [2] and everything else [3]. Common
> API is shared between C++ and .NET integrations. It is low-level interface
> to communicate with JVM (i.e. to perform forward and reverse JNI calls). It
> operates on pointers and JNI handles, and expects that you already
> serialized your objects in binary format amd put it to some memory chunk
> with certain layout. "Everything else" is high-level C++ API exposed to
> users.
> *6) Do not expect SWIG to help you a lot!*
> Taking all previous paragraphs in count - it is impossible to create
> valuable working implementation with SWIG. You will have to do a lot more
> than simple codegen.
> *7) Any other ways to integrate?*
> Yes, you can rely on our REST API [4]. We were planning to integrate with
> Node.JS this way - you just send HTTP requests to Ignite node(s). Every
> request is mapped to certain operation. However, this approach has two
> serious limitations:
> - It will be hard to employ advantages of binary interoperability
> - It will be very hard to implement server -> client callbacks which are of
> great importance for Ignite.
> As such you will have problems implementing compute API, continuous queries
> and so forth.
> Please let me know if you have any further questions.
> Vladimir.
> [1] https://issues.apache.org/jira/browse/IGNITE-3568
> [2]
> https://github.com/apache/ignite/tree/master/modules/platforms/cpp/common
> [3] https://github.com/apache/ignite/tree/master/modules/platforms/cpp
> [4] https://apacheignite.readme.io/docs/rest-api

View raw message