lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Commented: (LUCENE-1604) Stop creating huge arrays to represent the absense of field norms
Date Tue, 14 Apr 2009 20:11:14 GMT


Michael McCandless commented on LUCENE-1604:

bq. By the way, why everything surrounding norms map is heavily synchronized? I haven't found
a single write to the map outside of initialize().

You mean the SegmentReader.Norm class?  It's so only one thread attempts to load norms at
once.  And because cloned SegmentReaders share norms (incRef/decRef).  Also, the reader can
change norms (setNorm) which requires copy-on-write if more than one clone is sharing the
norms.  We'll need similar care for CSFs once they accept live changes, too.

bq. Somebody recently raised a topic on specialized Lucene

Right, that's LUCENE-1594.  I'll fold the norms/nonorms code gen into there....

Source code specialization gives sizable search performance gains w/ Lucene, but it'll be
quite a while before it's committable.  Hopefully as we pull some of the optimizations there
back into core (eg using random-access API to filter: LUCENE-1536), that gap shrinks.

Let's proceed for now with the null check in single-source scorer.  I'll run some perf tests
on it vs trunk, with the "norms present" case, to see where we stand.

> Stop creating huge arrays to represent the absense of field norms
> -----------------------------------------------------------------
>                 Key: LUCENE-1604
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.9
>            Reporter: Shon Vella
>            Priority: Minor
>             Fix For: 2.9
>         Attachments: LUCENE-1604.patch
> Creating and keeping around huge arrays that hold a constant value is very inefficient
both from a heap usage standpoint and from a localility of reference standpoint. It would
be much more efficient to use null to represent a missing norms table.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message