From dev-return-38055-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Tue Aug 21 11:15:37 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9FBF6180630 for ; Tue, 21 Aug 2018 11:15:36 +0200 (CEST) Received: (qmail 74165 invoked by uid 500); 21 Aug 2018 09:15:35 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 74153 invoked by uid 99); 21 Aug 2018 09:15:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Aug 2018 09:15:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id AC261C0115 for ; Tue, 21 Aug 2018 09:15:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.201 X-Spam-Level: * X-Spam-Status: No, score=1.201 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id bQN1PZpnzeG8 for ; Tue, 21 Aug 2018 09:15:33 +0000 (UTC) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id EA5C55F401 for ; Tue, 21 Aug 2018 09:15:32 +0000 (UTC) Received: by mail-wr1-f42.google.com with SMTP id v17-v6so595088wrr.9 for ; Tue, 21 Aug 2018 02:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=1zmKh7Ekn5VhynR5Jad1kWShzzUXKBVqnp0NQ3Ayvp4=; b=p6M8QAnuvBrSSHpA1suDdmTn10kr1hayPiKmSKSAt8lg4qhMYSPRNRcWC99AvGXOtg Od0uFbQ2CrbuCfGt2JHtzRlDyTHsTX3ICzvzW1EQRSeuHd4MTPGYtanvPLFMHx3fXVYc eM6vP+P6VCsI9wbv9pNYUOvONpGtdVtZrXByAULIfZdqQxDZzpsV9ImvVLSYTHhEl44P aUpFo/qK4cBgxgfocsM05iMbw/j6rko1NKzvQVz0RZtLquQsEYiypipGLjvpD01b/2hj 12MIKp0ZdsaAwgtxmjjqKIysEk3x6fzkKnnUGxFrvKIXDxaY9Qz7K8Erc4Cb6eALVM3y 6mwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=1zmKh7Ekn5VhynR5Jad1kWShzzUXKBVqnp0NQ3Ayvp4=; b=rXQfEs7CB0ZZvyH1FKAI1wD0zExGpRGaz6qlm3NehPF6+Qba3SASjAyLCTbI0DWBuh BjpXBxu9/VtyPSYooIXp2nOC33C3YgIg4VnWU6uQ4v48eNgnoHPMxVds5z289+8UU7Wv wKEjBispkxm2FY089xC6iPJuJOLraxJpmkVbxC6ExqqMEb/M5+fnmfXg1hzzWwpzAJNH 9gkinNxtVXIzBQmV48nmFhdRplnahTOjN1btlarQg5wdWfnAaBURmYisMLYN+UBb88j8 cs3/jF1WPCapXbRgHJj4foOQU1G6tK3N1A1PBCfIEVfce7TukRDRnDHbTfQ8n0Y7MJ8D 8Dgw== X-Gm-Message-State: AOUpUlH+ibeNqW6OXQsmeHNNmqg6Uvd+xu6OmKyGuC/dDy1mPgzUHYpP pINhwm0vimgzEWaDrfo2tEih44sHpFaw1/cJdUr7v2ss X-Google-Smtp-Source: AA+uWPxmL7CW+Mb9kX0UDayPRxnukUsSbmgPoY44cAhjHdbS2MFf/9RSaKi/oaU3nx7yDXapBGjMtpVreODyk6ESYJs= X-Received: by 2002:adf:bd90:: with SMTP id l16-v6mr32807601wrh.250.1534842931672; Tue, 21 Aug 2018 02:15:31 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Pavel Pereslegin Date: Tue, 21 Aug 2018 12:15:20 +0300 Message-ID: Subject: Re: Wrong off-heap size is reported for a node To: dev@ignite.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, Igniters. I assigned ticket [1] created by Denis and want to clarify how to log committed size. The metric offHeapSize (in DataRegionMetricsImpl) is always calculated, but getOffHeapSize returns zero if memory metrics are disabled for this data region. So I see the following options: 1. Modify method getOffHeapSize so that it always returns actual value offHeapSize. 2. Add another offHeapSize() method. 3. Output to log max size instead of committed (change "comm" to "max" in log output). 4. Don't bother about disabling metrics and output to log value returned by getOffHeapSize. Any thoughts? [1] https://issues.apache.org/jira/browse/IGNITE-9305 =D1=81=D0=B1, 18 =D0=B0=D0=B2=D0=B3. 2018 =D0=B3. =D0=B2 3:17, Denis Magda = : > > Vova, the things are even simpler - we have this > > ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the > number equal/comparabel to pageNumber X pageSize. > > > Igniters, if you believe that we need to do more work here then let's > do it iteratively. Let's fix the off-heap occupied size the way above > (just print out getPhysicalMemorySize() for every data region). Then > do the rest. This needs to be fixed in 2.7. > > > -- > > Denis > > > On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov > wrote: > > > Folks, > > > > We already have this: > > >>> PageMemory [pages=3D6997377] > > > > Then we can multiply it by page size and get occupied memory. Am I wron= g? > > > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov > > wrote: > > > > > Hi Maxim, > > > > > > thank you for stepping in and for finding these issues. Yes, these > > tickets > > > are correct. > > > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > > unassigned > > > if someone would like to implement this change. I will not have enoug= h > > time > > > to complete it in 1 month (before 2.7 release). > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > =D0=BF=D1=82, 17 =D0=B0=D0=B2=D0=B3. 2018 =D0=B3. =D0=B2 11:04, Maxim= Muzafarov : > > > > > > > Igniters, > > > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > > > memory > > > > metrics > > > > to new page memory semantics` and related previous disscustions to = it > > > [4]. > > > > > > > > Also we have some additional improvements to CacheMetrics: > > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > > CacheMetrics#getOffHeapAllocatedSize` > > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation an= d > > > fixes` > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > > [4] > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap= -memory-maximum-td17990.html > > > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > > > wrote: > > > > > > > > > Hi Igniters, > > > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.= 7. > > > > > > > > > > Offheap size is not reported by VM (it returns -1). To implement = it > > we > > > > need > > > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > =D0=BF=D1=82, 17 =D0=B0=D0=B2=D0=B3. 2018 =D0=B3. =D0=B2 6:10, De= nis Magda : > > > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > -- > > > > > > Denis > > > > > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > > > dsetrakyan@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda > > wrote: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't > > find > > > > out > > > > > > from > > > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager= ] > > > > Topology > > > > > > > > snapshot [ver=3D1, servers=3D1, clients=3D0, CPUs=3D64, > > *offheap=3D30.0GB*, > > > > > > > > heap=3D24.0GB] > > > > > > > > > > > > > > > > And that weird stuff was reported by the node (pay attentio= n to > > > the > > > > > > last > > > > > > > > line): > > > > > > > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_3= 1-Dec-2017] > > > > > > > > Metrics for local node (to disable set 'metricsLogFrequency= ' to > > > 0) > > > > > > > > ^-- Node [id=3Dc033026e, name=3Dcluster_31-Dec-2017, > > > > > > uptime=3D00:38:00.257] > > > > > > > > ^-- H/N/C [hosts=3D1, nodes=3D1, CPUs=3D64] > > > > > > > > ^-- CPU [cur=3D0.03%, avg=3D5.54%, GC=3D0%] > > > > > > > > ^-- PageMemory [pages=3D6997377] > > > > > > > > ^-- Heap [used=3D9706MB, free=3D61.18%, comm=3D22384MB] > > > > > > > > * ^-- Non heap [used=3D144MB, free=3D-1%, comm=3D148MB] = - this > > line > > > is > > > > > > > always > > > > > > > > the same!* > > > > > > > > > > > > > > > > Had to change the code by using > > > dataRegion.getPhysicalMemoryPages() > > > > > to > > > > > > > find > > > > > > > > out that actual off-heap usage size was > > > > > > > > >>> Physical Memory Size: 28651614208 =3D> 27324 MB, 26 GB > > > > > > > > > > > > > > > > Let's fix this issue in 2.7, I proposed a new format. Pleas= e > > > review > > > > > and > > > > > > > > share your thoughts: > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > > > > > -- > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- > > > > Maxim Muzafarov > > > > > > > > >