lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Sokolov <msoko...@gmail.com>
Subject Re: RamUsageCrawler
Date Thu, 06 Dec 2018 19:35:17 GMT
I agree, any attempt at improvement wouldn't be general. thanks for
the explanation.
On Thu, Dec 6, 2018 at 10:45 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
>
> I don't think it makes much sense, to be honest. Without actual
> reflection you're binding things to a particular implementation
> (you're assuming this and that JDK implementation). That's why we
> decided to remove it instead of making it overly complex (and possibly
> untrue). If a test is using it, perhaps the test itself should be
> fixed.
>
> D.
> On Thu, Dec 6, 2018 at 4:35 PM Michael Sokolov <msokolov@gmail.com> wrote:
> >
> > That's what it looked like to me, too. I wonder if it would be worth
> > improving the estimate for some very common Collections classes? I see
> > this comment eg in BaseIndexFileFormatTestCase:
> >
> >       // we have no way to estimate the size of these things in codecs although
> >       // something like a Collections.newSetFromMap(new HashMap<>()) uses
quite
> >       // some memory... So for now the test ignores the overhead of such
> >       // collections but can we do better?
> >
> > This is in testRamBytesUsed and there is a kind of fudge factor in
> > there for handling mismeasurement errors of the sort we are talking
> > about. Actually the test seems to be more about validating the
> > RamUsageTester than about validating the accounting in SegmentReader!
> > There are lots of other usages in tests, but I suppose they don't
> > require very precise handling of Collections classes (since they
> > pass)? Anyway it is certainly possible to improve the estimate quite a
> > bit and pretty easily for HashMap by simply counting the size of the
> > Node that is used for each entry, although given the dynamic nature of
> > these data structures (HashMap eg can use TreeNodes sometimes
> > depending on data distribution) it would be almost impossible to be
> > 100% accurate.
> > On Thu, Dec 6, 2018 at 7:14 AM Dawid Weiss <dawid.weiss@gmail.com> wrote:
> > >
> > > > It's entirely possible it fails to dig into Maps correctly with newer
Java
> > > > releases; maybe Dawid or Uwe would know?
> > >
> > > We have removed all reflection from that class a while ago exactly
> > > because of encapsulation issues introduced in newer Java versions.
> > >
> > > https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java
> > >
> > > I think you may be thinking of RamUsageTester which is in the test
> > > framework and indeed accumulates only keys and values from iterables.
> > > These methods are for tests only and are rough. You shouldn't rely on
> > > them for accurate memory consumption accounting (instead, use the
> > > Accountable interface).
> > >
> > > Dawid
> > >
> > >
> > > D.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message