ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seliverstov Igor <gvvinbl...@gmail.com>
Subject Re: [DISCUSSION] Urgent Ignite bug fix release
Date Mon, 28 Aug 2017 12:54:51 GMT
Ok, the check happens at the node start time or on NODE_JOIN event

in general it looks like:

1) calculate expected used memory = heap max + system cache max + all
custom policies max + default policy size and put it into a node attribute

2) get total physycal memory, calculate expected safe to be used memory
amount (leave 4 gb min or 20% of available memory for OS)

3) if expected used memory + expected used memory of other nodes on the
host > than safe to be used memory amount, start calculating suggestions

4) Each ignite instance needs at least 512mb heap + 40mb system cache +
100mb default polycy, if available memory is less we cannot suggest
anything reasonable, print warning, stop calculation.

5) check heap size (shouldn't exceed 30% of available memory (total_memory
- reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
calculated value or 512MB minimal)

6) check if system cache size changed, suggest default value if it's so

7) in case 100 mb * policies count < available memory, suggest using
default policy with max size equals to remaining memory (available - heap -
system cache)

8) calculate new size for each memory policy ( it's user defined size *
(remaining / (all_policies_size * nodes_cnt)); in proportion to
remaining memory, devided by nodes number on the host or 100 mb minimal)

9) print suggestions



2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:

> Igor, can you please describe the algorithm with all the thresholds?
>
> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinblade@gmail.com>
> wrote:
>
> > The suggestion here is based on initial settings, and it's so because
> there
> > is no other nodes on the host in the example.
> >
> > The algorithm tries to preserve the original ratio of memory policies
> > keeping numbers reasonable (for example after some thresshold it will
> > suggest not to use several memory policies if there is not enough of
> memory
> > for all of them) and taking into consideration nodes count on the host,
> > each jvm heap, needed memory for OS, etc
> >
> > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
> >
> > > Looks good, but why in the example provided are we suggesting 8GB? 2
> > nodes
> > > with 8GB will completely exhaust the available memory. I would suggest
> 6
> > or
> > > 7GB.
> > >
> > > Also, why 100MB for default policy. Anything under 1GB seems too small.
> > >
> > > Can you please comment?
> > >
> > > D.
> > >
> > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> gvvinblade@gmail.com>
> > > wrote:
> > >
> > > > One more example of possible warning:
> > > >
> > > > -----------------------------------------------------
> > > > Excessive memory usage by Ignite node process (performance may drop)
> > > > [requested=44613MB, available=15942MB].
> > > >
> > > > Please tune the folowing settings as suggested:
> > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > >   MemoryPolicyConfiguration.maxSize     for bigPlc: 8102MB
> > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > >   MemoryPolicyConfiguration.maxSize     for dfltPlc: 100MB
> > > >
> > > > Current settings:
> > > >   Java Heap  maxSize: 3543MB
> > > >   Java Heap initSize: 250MB
> > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > >   MemoryPolicyConfiguration.maxSize     for bigPlc: 40960MB
> > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > >   MemoryPolicyConfiguration.maxSize     for dfltPlc: 10MB
> > > >   The overall expected memory usage by all Ignite nodes on the host:
> > > > 44613MB
> > > > -----------------------------------------------------
> > > >
> > > > Your thoughts?
> > > >
> > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dmagda@apache.org>:
> > > >
> > > > > Guys,
> > > > >
> > > > > ML lib profile is missing in 2.1 release! That must be fixed and
> > rolled
> > > > > out in this emergency release:
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > > >
> > > > > Oleg, Yuri, please step in and handle the issue.
> > > > >
> > > > > BTW, who is considered to be the release manager of this release?
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > I like the format proposed by Denis, very clear.
> > > > > >
> > > > > > However, I also do not understand why a user should change the
> size
> > > of
> > > > > some
> > > > > > system cache. How would a user ever know what value to put there?
> > > This
> > > > > > value should be configured by Ignite automatically.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dmagda@apache.org>
> > > > wrote:
> > > > > >
> > > > > >> Igor,
> > > > > >>
> > > > > >> Let me suggest this format.
> > > > > >>
> > > > > >> ---------------------------------------------
> > > > > >> Excessive memory usage by Ignite node process (performance
may
> > drop)
> > > > > >> [requested=29251MB, available=15942MB]
> > > > > >>
> > > > > >> Please tune the following settings:
> > > > > >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested
> value]
> > > > > >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> > > > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1}
=
> > suggested
> > > > > >> value]
> > > > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2}
=
> > suggested
> > > > > >> value]
> > > > > >>
> > > > > >> Current settings:
> > > > > >>   [DefaultMemoryPolicySize = value]
> > > > > >>   [{policy_name_1} size = value]
> > > > > >>   [{policy_name_1} size = value]
> > > > > >>   SystemCacheInitialSize = value
> > > > > >>   SystemCacheMaxSize = value
> > > > > >>   Java Heap Init Size = value
> > > > > >>   Java Heap Max Size = value
> > > > > >>
> > > > > >> The overall memory usage by all Ignite nodes on the host:
value
> > > > > >> -------------------------------------------
> > > > > >>
> > > > > >> Records in […] are optional. If custom memory policy is
not set
> or
> > > the
> > > > > >> default memory policy is overridden the output will miss
some of
> > the
> > > > > rows.
> > > > > >>
> > > > > >> As for systemCacheMaxSize, it should be show ONLY if the
> parameter
> > > was
> > > > > set
> > > > > >> explicitly by user code. Otherwise, the platform should
be wise
> > > enough
> > > > > to
> > > > > >> instantiate it properly depending on the host memory usage.
> > > > > >>
> > > > > >> —
> > > > > >> Denis
> > > > > >>
> > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > > gvvinblade@gmail.com>
> > > > > >> wrote:
> > > > > >>>
> > > > > >>> The message without logging layout:
> > > > > >>>
> > > > > >>> Not enough memory for current process [required=29251MB,
> > > > > >> available=15942MB].
> > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize
and
> > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease
memory
> > > > > allocated
> > > > > >>> for each node.
> > > > > >>>
> > > > > >>> Current settings:
> > > > > >>>  HeapInit=250MB
> > > > > >>>  HeapMax=3543MB
> > > > > >>>  DefaultMemoryPolicySize=12753MB
> > > > > >>>  SystemCacheInitialSize=40MB
> > > > > >>>  SystemCacheMaxSize=100MB
> > > > > >>>
> > > > > >>> Other ignite instanses on the server require: 12853MB
> > > > > >>>
> > > > > >>> I think it's make sense to describe what these numbers
are
> > consist
> > > > of.
> > > > > >>>
> > > > > >>> We simple say which parameters have an impact on how
much
> memory
> > > the
> > > > > >>> instance needs and their (parameters) actual values.
> > > > > >>>
> > > > > >>> Also we notice that more than one Ignite instance are
ran on
> the
> > > > server
> > > > > >> or
> > > > > >>> workstation and it also consumes memory.
> > > > > >>>
> > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy
Setrakyan" <
> > > > > >>> dsetrakyan@apache.org> написал:
> > > > > >>>
> > > > > >>>> Igor, what is this flood of WARN messaging coming
after the
> > text?
> > > > Are
> > > > > we
> > > > > >>>> really going to print this whole thing out?
> > > > > >>>>
> > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor
<
> > > > > gvvinblade@gmail.com
> > > > > >>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> This message appears on topology change in case
the available
> > > > memory
> > > > > is
> > > > > >>>>> exceeded
> > > > > >>>>>
> > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor
<
> > > gvvinblade@gmail.com
> > > > >:
> > > > > >>>>>
> > > > > >>>>>> An example of current impl:
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
Not enough
> > memory
> > > > for
> > > > > >>>>>> current process [required=29251MB, available=15942MB].
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
Please change
> > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and
> > MemoryConfiguration.
> > > > > >>>>> defaultMemoryPolicySize
> > > > > >>>>>> to decrease memory allocated for each node.
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
Current
> > settings:
> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >  HeapInit=250MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> >  HeapMax=3543MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>> DefaultMemoryPolicySize=
> > > > > >>>>>> 12753MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> SystemCacheInitialSize=40MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>> SystemCacheMaxSize=100MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
Other ignite
> > > > > instanses
> > > > > >>>> on
> > > > > >>>>>> the server require: 12853MB
> > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%
> > > internal.
> > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager]
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov
<
> > skozlov@gridgain.com
> > > >:
> > > > > >>>>>>
> > > > > >>>>>>> I suppose we should not forget JVM heap
size and suggest to
> > > > reduce
> > > > > >>>> both
> > > > > >>>>>>> options
> > > > > >>>>>>>
> > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy
Setrakyan <
> > > > > >>>>> dsetrakyan@apache.org
> > > > > >>>>>>>>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Igor, I would change the message.
How about this:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Required RAM size is larger than
total physical memory
> > > available
> > > > > for
> > > > > >>>>> OS.
> > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty
and
> > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty
to decrease
> memory
> > > > > >>>>> allocated
> > > > > >>>>>>> for
> > > > > >>>>>>>>> each node.
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> Also, can we calculate what the
memory size allocated for
> > each
> > > > > node
> > > > > >>>>>>> should
> > > > > >>>>>>>> be? In that case we should suggest
it.
> > > > > >>>>>>>>
> > > > > >>>>>>>> D.
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM,
Seliverstov Igor <
> > > > > >>>>> gvvinblade@gmail.com
> > > > > >>>>>>>>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> What do you, guys think about
next warning?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> System cache's MemoryPolicy
size is configured to 40 MB.
> > Use
> > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize
property to
> > change
> > > > the
> > > > > >>>>>>>> setting.
> > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>>>> Required RAM size
is larger than total physical memory
> > > > > >>>> available
> > > > > >>>>>>> for
> > > > > >>>>>>>>> OS.
> > > > > >>>>>>>>>>>> Check your configuration
to avoid swap partition
> usage.
> > > > > >>>>>>>>>>>> Use MemoryConfiguration
and MemoryPolicyConfiguration
> to
> > > > > >>>> change
> > > > > >>>>>>> the
> > > > > >>>>>>>>> settings.
> > > > > >>>>>>>>>>>> Physical memory
[required=16397MB, available=15942MB]
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> Peer class loading is enabled
(disable it in production
> for
> > > > > >>>>>>> performance
> > > > > >>>>>>>> and
> > > > > >>>>>>>>> deployment consistency reasons)
> > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> Configured caches [in 'sysMemPlc'
memoryPolicy:
> > > > > >>>>> ['ignite-sys-cache']]
> > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO
> > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][
> > GridHomePat
> > > > > >>>>>>> hSelfTest0]
> > > > > >>>>>>>>> 3-rd party licenses can be found
at:
> > > > > >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/
> > > > licenses
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov
Zhdanov <
> > > yzhdanov@apache.org
> > > > >:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Agree, let's release new
version including tickets
> > mentioned
> > > > by
> > > > > >>>>>>> Denis
> > > > > >>>>>>>> and
> > > > > >>>>>>>>>> Ivan.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> --Yakov
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> --
> > > > > >>>>>>> Sergey Kozlov
> > > > > >>>>>>> GridGain Systems
> > > > > >>>>>>> www.gridgain.com
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message