ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Plehanov <plehanov.a...@gmail.com>
Subject Re: Wrong off-heap size is reported for a node
Date Wed, 12 Sep 2018 19:33:54 GMT
Hi Dmitriy,

Yes, I will finish review tomorrow.

ср, 12 сент. 2018 г. в 20:37, Dmitriy Pavlov <dpavlov.spb@gmail.com>:

> Hi Igniters,
>
> It seems the issue is a blocker of 2.7.
>
> We need a volunteer here to run review.
>
> Alexey P. will you step in?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 29 авг. 2018 г. в 16:16, Nikita Amelchev <nsamelchev@gmail.com>:
>
> > Pavel,
> >
> > I think that point 1 is the correct way to calculate the committed
> > size for a log. It is already calculated regardless of the
> > metricsEnabled flag.
> >
> > In addition, I suggest more readable log format for data regions in
> > issue comments [1].
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9305
> >
> > вт, 21 авг. 2018 г. в 12:15, Pavel Pereslegin <xxtern@gmail.com>:
> > >
> > > Hello, Igniters.
> > >
> > > I assigned ticket [1] created by Denis and want to clarify how to log
> > > committed size.
> > > The metric offHeapSize (in DataRegionMetricsImpl) is always
> > > calculated, but getOffHeapSize returns zero if memory metrics are
> > > disabled for this data region.
> > >
> > > So I see the following options:
> > > 1. Modify method getOffHeapSize so that it always returns actual value
> > > offHeapSize.
> > > 2. Add another offHeapSize() method.
> > > 3. Output to log max size instead of committed (change "comm" to "max"
> > > in log output).
> > > 4. Don't bother about disabling metrics and output to log value
> > > returned by getOffHeapSize.
> > >
> > > Any thoughts?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9305
> > > сб, 18 авг. 2018 г. в 3:17, Denis Magda <dmagda@apache.org>:
> > > >
> > > > Vova, the things are even simpler - we have this
> > > >
> > > > ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the
> > > > number equal/comparabel to pageNumber X pageSize.
> > > >
> > > >
> > > > Igniters, if you believe that we need to do more work here then let's
> > > > do it iteratively. Let's fix the off-heap occupied size the way above
> > > > (just print out getPhysicalMemorySize() for every data region). Then
> > > > do the rest. This needs to be fixed in 2.7.
> > > >
> > > >
> > > > --
> > > >
> > > > Denis
> > > >
> > > >
> > > > On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov <
> vozerov@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > We already have this:
> > > > > >>> PageMemory [pages=6997377]
> > > > >
> > > > > Then we can multiply it by page size and get occupied memory. Am
I
> > wrong?
> > > > >
> > > > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Maxim,
> > > > > >
> > > > > > thank you for stepping in and for finding these issues. Yes,
> these
> > > > > tickets
> > > > > > are correct.
> > > > > >
> > > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583
to
> > > > > unassigned
> > > > > > if someone would like to implement this change. I will not have
> > enough
> > > > > time
> > > > > > to complete it in 1 month (before 2.7 release).
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov <maxmuzaf@gmail.com
> >:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch
> > non-heap
> > > > > > memory
> > > > > > > metrics
> > > > > > > to new page memory semantics` and related previous disscustions
> > to it
> > > > > > [4].
> > > > > > >
> > > > > > > Also we have some additional improvements to CacheMetrics:
> > > > > > > IGNITE-5490 [2] - `Implement replacement for obsolete
> > > > > > > CacheMetrics#getOffHeapAllocatedSize`
> > > > > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup,
> documentation
> > and
> > > > > > fixes`
> > > > > > >
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583
> > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490
> > > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765
> > > > > > > [4]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html
> > > > > > >
> > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov <
> > dpavlov.spb@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > It is not an easy fix, so I'm not sure it is possible
to do
> in
> > 2.7.
> > > > > > > >
> > > > > > > > Offheap size is not reported by VM (it returns -1).
To
> > implement it
> > > > > we
> > > > > > > need
> > > > > > > > totally migrate off-heap memory metrics to durable
memory
> data.
> > > > > > > >
> > > > > > > > I think this issue was reported and I'll find the
duplicate.
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda <dmagda@apache.org>:
> > > > > > > >
> > > > > > > > > Yes, it was at the end of my wordy email :)
> > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan
<
> > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Is there a blocker ticket for 2.7?
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda
<
> dmagda@apache.org
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Igniters,
> > > > > > > > > > >
> > > > > > > > > > > Was troubleshooting an Ignite deployment
today and
> > couldn't
> > > > > find
> > > > > > > out
> > > > > > > > > from
> > > > > > > > > > > the logs what was the actual off-heap
space used.
> > > > > > > > > > >
> > > > > > > > > > > Those were the given memory resoures
(Ignite 2.6):
> > > > > > > > > > >
> > > > > > > > > > > [2018-08-16 15:07:49,961][INFO
> > ][main][GridDiscoveryManager]
> > > > > > > Topology
> > > > > > > > > > > snapshot [ver=1, servers=1, clients=0,
CPUs=64,
> > > > > *offheap=30.0GB*,
> > > > > > > > > > > heap=24.0GB]
> > > > > > > > > > >
> > > > > > > > > > > And that weird stuff was reported by
the node (pay
> > attention to
> > > > > > the
> > > > > > > > > last
> > > > > > > > > > > line):
> > > > > > > > > > >
> > > > > > > > > > > [2018-08-16 15:45:50,211][INFO
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
> > > > > > > > > > > Metrics for local node (to disable
set
> > 'metricsLogFrequency' to
> > > > > > 0)
> > > > > > > > > > >     ^-- Node [id=c033026e, name=cluster_31-Dec-2017,
> > > > > > > > > uptime=00:38:00.257]
> > > > > > > > > > >     ^-- H/N/C [hosts=1, nodes=1, CPUs=64]
> > > > > > > > > > >     ^-- CPU [cur=0.03%, avg=5.54%,
GC=0%]
> > > > > > > > > > >     ^-- PageMemory [pages=6997377]
> > > > > > > > > > >     ^-- Heap [used=9706MB, free=61.18%,
comm=22384MB]
> > > > > > > > > > >    * ^-- Non heap [used=144MB, free=-1%,
comm=148MB] -
> > this
> > > > > line
> > > > > > is
> > > > > > > > > > always
> > > > > > > > > > > the same!*
> > > > > > > > > > >
> > > > > > > > > > > Had to change the code by using
> > > > > > dataRegion.getPhysicalMemoryPages()
> > > > > > > > to
> > > > > > > > > > find
> > > > > > > > > > > out that actual off-heap usage size
was
> > > > > > > > > > > >>> Physical Memory Size:
28651614208 => 27324 MB, 26
> GB
> > > > > > > > > > >
> > > > > > > > > > > Let's fix this issue in 2.7, I proposed
a new format.
> > Please
> > > > > > review
> > > > > > > > and
> > > > > > > > > > > share your thoughts:
> > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Denis
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > --
> > > > > > > Maxim Muzafarov
> > > > > > >
> > > > > >
> > > > >
> >
> >
> >
> > --
> > Best wishes,
> > Amelchev Nikita
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message