ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Muzafarov <maxmu...@gmail.com>
Subject [DISCUSSION] Channel communication between nodes
Date Fri, 01 Mar 2019 16:43:58 GMT
Igniters,

Apache Ignite has a very suitable messaging user interface [1] for
topic-based communication between nodes (or a specific group of nodes
within a cluster). The messaging functionality in Ignite is provided
via IgniteMessaging interface. It allows:
- send a message to a certain topic
- register local\remote listeners

I really like this feature, but the disadvantage here is when the user
wants to transfer a large amount of binary data (e.g. files) between
nodes he must create a complex logic to wrap it into messages. I think
Ignite could have an interface e.g. IgniteChannels which will allow:
- register local\remote listeners for channel created\destroy events.
- create a channel connection (a wrapped socket channel) to a certain
node\group of nodes and the desired topic

As another suitable case where such a feature can be applied is
internal usage for Apache Ignite needs. I can mention here the task of
cluster rebalancing by sending cache partition files between nodes.
I've posted a small description of it on the IEP-28 page [2].


WDYT about it?

---

API (assumed)

IgniteChannels chnls = ignite0.channels();
chnls.remoteListen(TOPIC.MY_TOPIC, new RemoteListener());

IgniteSocketChannel ch0 = chnls.channel(node, TOPIC.MY_TOPIC);
ch0.writeInt(bigFile.size());
ch0.transferTo(FileChannel.open(bigFile.path(), StandardOpenOption.READ))


/** */

private class RemoteListener
    implements IgniteBiPredicate<UUID, IgniteSocketChannel> {

    @IgniteInstanceResource
    private Ignite ignite;

    @Override public boolean apply(
        UUID nodeId,
        IgniteSocketChannel ch
    ) {
        int size = ch.readInt();
        ignite.fileSystem("base")
            .create("bigfile.mpg")
            .transferFrom(ch, size);
        return true;
    }
}


[1] https://apacheignite.readme.io/docs/messaging
[2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-CommunicationSpi

Mime
View raw message