From java-user-return-64124-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Thu Dec 6 20:43:56 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6A32C180674 for ; Thu, 6 Dec 2018 20:43:55 +0100 (CET) Received: (qmail 76085 invoked by uid 500); 6 Dec 2018 19:43:49 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 76072 invoked by uid 99); 6 Dec 2018 19:43:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Dec 2018 19:43:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 33942180CC6 for ; Thu, 6 Dec 2018 19:43:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.597 X-Spam-Level: X-Spam-Status: No, score=0.597 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 0ekA7c1x6u7V for ; Thu, 6 Dec 2018 19:43:47 +0000 (UTC) Received: from mail-it1-f174.google.com (mail-it1-f174.google.com [209.85.166.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 72CE45FB07 for ; Thu, 6 Dec 2018 19:35:30 +0000 (UTC) Received: by mail-it1-f174.google.com with SMTP id m8so17732616itk.0 for ; Thu, 06 Dec 2018 11:35:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=LLeGU2bEIA1Xop8FrdJS2bLye6MIT1myJS9dB461Qxg=; b=LHNWNqpAxGqdcocC6BN4UuTlR5eo5tlfxHTlAPJ0UaLPW+G2F3U/iR3U2dR8kl2iQr GVlpZ+yWRL9JHz8QZ8AJ8X0iTKdRKKiAGUkyfarZ2Z+zrtXdPYTXE+UaUVtwAZ4KSHws H5OCV1HQfwSLBfBKw5pO/taIYnKj+LfCSvhxNLZY1tS+kZtLuxSyC4OwN/+loCHINZTn g+UsGpkpjgdLALg3Wo92iE+KnmA/qgSsgbC6uVYtMocEQOXteE6ey+ywqS9wYYC1s1B6 72vxHFQnplIp5V8FsJjJZ6MnBok0jWCQQURuE0++d3w7ENUg7PkIRIv2uWnFfLMNKi8z 1J2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=LLeGU2bEIA1Xop8FrdJS2bLye6MIT1myJS9dB461Qxg=; b=Twoyjyq0aLbplw/NvquSd5EgLGM24T6u+DmcF83IHqwMOvomUwA5xU9x3BOMW/Evcp B/dcieG/iwOUkZHqr/jRuNegbYx4SBKWzZXxfompbAfnqoF8SBzAvBMLkcdMLrZprfj8 PBfh+amSwgMTEmzrH81WoCDUUDojerKnpcBUwHf8j+2NKdUOKkQ3BD2tKoew8G50sJ2i x6K4UXVGhCsXT1pLMom/tQIGisJhAdsYlZeHgff5k6gjipU+OYSN7HKcpvElCMsAukMX /7I9+IK0zJ6U22x/kTa+LkC1WYqa1rqI0DO/2Dbh85mtlIEYLeSr2rovaalU+TkVvdLk Sp0w== X-Gm-Message-State: AA+aEWbr26dTQj3eFqroR134DVCBAzn8xXc3nDVkwLyOJRvnrv5uxdTc DcPlVYtVxEPNvGrKG/1VcU5tvqTQnEB7fdKOAiN0Iw== X-Google-Smtp-Source: AFSGD/VSgFp1s/pCSQEorQc/3TZOwLWIw+4RPYgB1rIWv9N7iPsAnesK+atP9dOYx4kTfIVsMM/1xFMHAy7jlcMyR7U= X-Received: by 2002:a24:b64a:: with SMTP id d10mr13730248itj.149.1544124929585; Thu, 06 Dec 2018 11:35:29 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Michael Sokolov Date: Thu, 6 Dec 2018 14:35:17 -0500 Message-ID: Subject: Re: RamUsageCrawler To: java-user@lucene.apache.org Content-Type: text/plain; charset="UTF-8" I agree, any attempt at improvement wouldn't be general. thanks for the explanation. On Thu, Dec 6, 2018 at 10:45 AM Dawid Weiss wrote: > > I don't think it makes much sense, to be honest. Without actual > reflection you're binding things to a particular implementation > (you're assuming this and that JDK implementation). That's why we > decided to remove it instead of making it overly complex (and possibly > untrue). If a test is using it, perhaps the test itself should be > fixed. > > D. > On Thu, Dec 6, 2018 at 4:35 PM Michael Sokolov wrote: > > > > That's what it looked like to me, too. I wonder if it would be worth > > improving the estimate for some very common Collections classes? I see > > this comment eg in BaseIndexFileFormatTestCase: > > > > // we have no way to estimate the size of these things in codecs although > > // something like a Collections.newSetFromMap(new HashMap<>()) uses quite > > // some memory... So for now the test ignores the overhead of such > > // collections but can we do better? > > > > This is in testRamBytesUsed and there is a kind of fudge factor in > > there for handling mismeasurement errors of the sort we are talking > > about. Actually the test seems to be more about validating the > > RamUsageTester than about validating the accounting in SegmentReader! > > There are lots of other usages in tests, but I suppose they don't > > require very precise handling of Collections classes (since they > > pass)? Anyway it is certainly possible to improve the estimate quite a > > bit and pretty easily for HashMap by simply counting the size of the > > Node that is used for each entry, although given the dynamic nature of > > these data structures (HashMap eg can use TreeNodes sometimes > > depending on data distribution) it would be almost impossible to be > > 100% accurate. > > On Thu, Dec 6, 2018 at 7:14 AM Dawid Weiss wrote: > > > > > > > It's entirely possible it fails to dig into Maps correctly with newer Java > > > > releases; maybe Dawid or Uwe would know? > > > > > > We have removed all reflection from that class a while ago exactly > > > because of encapsulation issues introduced in newer Java versions. > > > > > > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java > > > > > > I think you may be thinking of RamUsageTester which is in the test > > > framework and indeed accumulates only keys and values from iterables. > > > These methods are for tests only and are rough. You shouldn't rely on > > > them for accurate memory consumption accounting (instead, use the > > > Accountable interface). > > > > > > Dawid > > > > > > > > > D. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > > > For additional commands, e-mail: java-user-help@lucene.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > > For additional commands, e-mail: java-user-help@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org