ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Павлухин Иван <vololo...@gmail.com>
Subject Re: Pre-touch for Ignite off-heap memory
Date Wed, 12 Dec 2018 19:14:11 GMT
Hi Stan,

As I participated in discussion in current thread I would like to
leave a comment.

Your concerns looks clear for me and if you believe that pre-touch
will help product users then I have nothing against it.
ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov <nizhikov@apache.org>:
>
> Hello, Stanislav.
>
> As far as I can see, we have controversal ticket based on previous dev-list discussion:
>
> IGNITE-9113 - Allocate memory for a data region when first cache assigned to this region
is created [1]
>
> I planned to implement it soon.
> Looks like we should have several options of memory(data regions) allocation:
>
>         - allocate all on startup (AFAIK this is how current implementation behave)
>         - allocate all on startup AND pre touch.
>         - allocate specific data region for first assignment.
>         - allocate specific data region for first assignment AND pre touch.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > Igniters,
> >
> > What is being suggested here is an Ignite off-heap’s version of Java’s -XX:+AlwaysPreTouch.
> > The latter is known to be used to guarantee that the committed memory is backed
by physical RAM.
> > This ensures that
> > a) JVM doesn’t have to do it during the actual work (avoiding overhead for physical
page allocation, possible contention with page cache, etc)
> > b) JVM fails fast if the Xmx is greater than available RAM
> > c) New processes will not be able to claim the memory JVM took for itself
> >
> > Currently one can’t get the same benefits for Ignite because we use off-heap as
well as heap.
> > So, we can implement a similar feature for Ignite – and make sure the users can
get all the memory pre-touch benefits if they want.
> > Of course, it impacts startup time so we should enable it by a configuration property
(I’d suggest a system property for now).
> >
> > Are there any objections to implementing this?
> >
> > Thanks,
> > Stan
> >
> > From: Павлухин Иван
> > Sent: 31 октября 2018 г. 12:50
> > To: dev@ignite.apache.org
> > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> > Hi,
> >
> > I did an experiment described above. I created a patch with pre-touch [1]
> > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > Ignite with equal values for initial and max sizes for each data region.
> > I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> > to observe JVM OOM-killed.
> >
> > [1] https://github.com/apache/ignite/pull/5220
> > [2]
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid5763-log
> > [3]
> > https://gist.github.com/pavlukhin/e5e6605e9b43666266667ba8d1aab42f#file-hs_err_pid6411-log
> >
> > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван <vololo100@gmail.com>:
> >
> > > Hi guys,
> > >
> > > I am not aware that it is possible to run JVM in "allocation-free" fashion.
> > > If you know that it is possible please share it. As I know JVM allocates
> > > memory out of garbage collectable area for internal purposes like JIT,
> > > GC itself. Also native built-it code can request memory allocation from OS,
> > > e.g. zip facilities. If we imagine that system is running under memory
> > > allocation
> > > which is close to a limit then even small allocation request can fail and
> > > lead
> > > to OOM killing.
> > >
> > > But I think that a simple and useful thing that could be done first is
> > > making
> > > an experiment. As Andrey mentioned
> > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > >
> > > region
> > > > MAX size equal to MIN and get region allocated on node start.
> > >
> > > If it is so then it should not be hard to launch Ignite and observe it
> > > running
> > > very close to OS memory limit with Java heap and Ignite off-heap
> > > pre-touched.
> > > After that one could check whether it is possible to observe Ignite OOM
> > > killed.
> > > Let's say my bet is that it is relatively easy to catch OOM here.
> > >
> > > What do you think?
> > >
> > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov <yzhdanov@apache.org>:
> > >
> > > > Andrey,
> > > >
> > > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > > What
> > > > do you mean by JVM internal needs? In my understanding if user enables
> > > > option to pretouch heap and fixes the heap to prevent jvm releasing memory
> > > > back to OS, then OOM killing is very unlikely.
> > > >
> > > > I would agree that pretouch for offheap may be helpful in many cases.
> > > >
> > > > --Yakov
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
> >



-- 
Best regards,
Ivan Pavlukhin

Mime
View raw message