ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alpha centauri <helium...@hotmail.com>
Subject RE: From one cache to another
Date Sun, 06 Dec 2015 00:51:27 GMT

Hello,

Thanks for your answers, guys.

It works now but I struggled a lot with the serialization process.

Maybe somewhere strongly underline the fact in the documentation that ONLY certain type of
classes (I think it's closures and some other
inner tasks) are part of the zero deployment mechanisms. For like two hours, I didn't understood
why any object inside my tasks would not be automagically deployed on the cluster nodes.

It is still a bit annoying. Once I create a new processor, I have to restart all my nodes
to have it properly deployed everywhere. Is there something I'm missing ?

Anyway thanks, Ignite is a really powerfull thing!



Date: Fri, 27 Nov 2015 13:55:08 +0300
Subject: Re: From one cache to another
From: agura@gridgain.com
To: user@ignite.apache.org

Hi,

You should not serialize cache instances. Actually your cache is already created on all nodes
of your cluster and you can obtain it using ignite.cache(<name>) or ignite.getOrCreateCache(<name>)
methods.

I assume that you try to start Ignite instance with default name from your ComplexObjectProcessor
but Ignite is started already. You can declare Ignite reference in ComplexObjectProcessor
and annotate it with @IgniteInstanceResource. When your processor will be deserialized on
remote node Ignite instance will be injected automatically. It looks like this:

public class ComplexObjectProcessor {

    @IgniteInstanceResource    private Ignite ignite

    ...
}

Make sure that your ComplexObjectProcessor is regular class or inner static class. Non-static
inner class instances have reference to enclosing class that will be serialized with inner
class.


On Thu, Nov 26, 2015 at 9:08 PM, alpha centauri <helium667@hotmail.com> wrote:



Hello,

I'm a new Apache Ignite User. I'm trying to prototype a simple use case : 
1. create two caches
2. read strings from a file and put them in the first cache
3. compute strings into complex objects put the complex objects into second cache

My code looks like this : 

// Part 1
final IgniteCache<Integer, String> cacheString = ignite.getOrCreateCache("somestrings");
final IgniteCache<String, SomeObjects> cache = ignite.getOrCreateCache("someobjects");

// Part 2
try (IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer(cacheString.getName()))
{
                stmr.allowOverwrite(true);
                try (BufferedReader br = Files.newReader(file, StandardCharsets.UTF_8)) {
                    for (String line; (line = br.readLine()) != null; ) {
                        if (line !=null && line.length() > 0) {
                            stmr.addData(index, line);
                        }
                    }

                }
            }

// Part 3 is where I struggle and need help
IgniteCompute compute = ignite.compute();

for (int i = 0; i < cacheString.size(); i++) {
         final int key = i;
         compute.affinityRun(cacheString.getName(), key, new ComplexObjectProcessor(cache,
cacheString.get(key)));
}

The issue is that the ComplexObjectProcessor is serialized to my grid nodes and if I put the
destination cache, it will crash because the cache cannot be serialized. If I initialize the
cache from my ComplexObjectProcessor, it will complain about the fact that Ignite has already
been instanciated  outside. ComplexObjectProcessor has a lot of code so an inside/inline class
is not an option.
The cacheString contains 100Millions lines so I really want to split the complex object processor
with compute tasks.

Any solution? Or code sample would greatly help.

Thks ahead,
AC

 		 	   		  


-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com
 		 	   		  
Mime
View raw message