ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: inflated cache size
Date Sun, 29 Oct 2017 08:02:13 GMT
Probably you have a RDD with Java objects which consume a huge amount of memory. If you use
RDD you can try Kyroserializer which save memory and may even be faster.

> On 29. Oct 2017, at 08:23, Yair Ogen <yairogen@gmail.com> wrote:
> 
> Hi,
> 
> I'm trying out the ignite-spark support. I have a dataframe that was created from reading
a csv file sized around 800MB.
> 
> It seems that When I store the rdd from this dataframe in ignite using saveValues api
in IgniteContext it takes around 2GB of RAM.
> 
> Naturally once we add more dataframes, joins and computations we get OOM errors even
though we have more than enough RAM.
> 
> Any ideas why the inflated memory?
> 
> Attached is my config.
> 
> Yair
> <ignite-config.xml>

Mime
View raw message