ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vkulichenko <valentin.kuliche...@gmail.com>
Subject Re: Huge heap size for ignite cache when storing 20 million objects of small size
Date Wed, 17 Feb 2016 05:02:34 GMT
Hi vidhu,

First of all, can you please subscribe to the mailing list so that community
receives email notifications? Please follow this simple instruction:

vidhu wrote
> I am facing a problem where my heap reaches to about 2.8 gb when I am
> storing 20 million objects. Size  of each object is approx. 462 bytes.
> Structure of my object is a simple pojo having 101 fields each of int
> type.
> I tried using following 3 marshalling techniques:
> 1.) Default marshaller of Ignite. (Size of heap ~ 2.8 gb, looking with
> Visual VM find that most of objects on heap are byte array.)
> 2.) Optimized marshaller. (Again size was ~ 2.8 gb, looking with Visual VM
> find that this time size of objects on heap of byte array is half + there
> are lot of objects of class that I put in cache. )
> 3.) Tried implementing Binarylizable in my class that I am storing in
> cache. I used rawWriter. This time size reduced considerably but I was not
> able to read values out of cache, (getting null on read). 
> I am implementing this in java version 1.8.

20,000,000 * 462 bytes is already around 8.6GB of memory, so 2.8 is not
enough even without Ignite's overhead. Did you try to allocate more heap
memory and/or start more nodes? With your sizings I would start with two
nodes, 8GB of memory per node.

As for options you tried, I would recommend to use default binary marshaller
because it's much more compact. Also you should take a look at offheap
memory [1]. It can be useful if you're going to have a lot of data.

[1] https://apacheignite.readme.io/docs/off-heap-memory


View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3050.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message