ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Wilson <raymond_wil...@trimble.com>
Subject Running out of memory using ScanQuery. Memory leak?
Date Wed, 18 Oct 2017 00:01:58 GMT
I have run into something odd using ScanQuery (using Ignite 2.2, C# client
and native persistence). I have a fairly simple piece of scanning code
below, it simple looks through all the keys in a cache and emits them to a
file.



It creates a ScanQuery, sets its page size to 1, and explicitly disposes of
each value once it’s key has been written out (to ensure there are no .Net
GC nasties). SubGridSpatialAffinityKey is a simple struct with a few value
fields in it. This code is running in the context of a client node and is
asking other server nodes for the keys via the ScanQuery.



        private void writeKeysSpatial(string title, StreamWriter writer,
ICache<SubGridSpatialAffinityKey, MemoryStream> cache)

        {

            writer.WriteLine(title);



            var scanQuery = new ScanQuery<SubGridSpatialAffinityKey,
MemoryStream>();

            scanQuery.PageSize = 1; // Restrict the number of keys
requested in each page to reduce memory consumption



            IQueryCursor<ICacheEntry<SubGridSpatialAffinityKey, MemoryStream>>
queryCursor = cache.Query(scanQuery);



            foreach (ICacheEntry<SubGridSpatialAffinityKey, MemoryStream>
cacheEntry in queryCursor)

            {

                writer.WriteLine(cacheEntry.Key.ToString());

                cacheEntry.Value.Dispose();

            }

        }



When I run this code it progresses for quite a while before running into a
crisis in the JVM side of things as seen in the snap from JVisualVM below.
Once it reaches this state no further progress is made, and code execution
hangs inside the foreach loop. My impression here is that the memory
requirements through the process of the scan query should be fairly steady
state, but this does not seem to be the case. The total amount of data held
in the cache being scanned is less than 1 Gb.





Raymond.

Mime
View raw message