lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <paul.elsc...@xs4all.nl>
Subject Re: Eliminating norms ... completley
Date Sat, 08 Oct 2005 08:54:13 GMT

On Friday 07 October 2005 23:18, Chris Hostetter wrote:
> 
...
> So I'm curious what you guys think...
> 
>   1) Regarding the root problem: is there any other things you can think
>      of besides norms[] that would contribute to the memory foot print
>      needed by a large number of indexed fields?

One could use further compressing and/or memory mapped arrays,
but these are not as easy as ignoring the norms on disk altogether.

>   2) Can you think of a clean way for individual applications to eliminate
>      norms (via subclassing the lucene code base - ie: no patching)

See also this thread on auto-filters and boolean fields:
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg08007.html

Treating a field as boolean would involve adding a TermScorer that ignores
the field norms. Everything that uses that could also be added, so, in
principle, no patching is needed. At some point, the knowledge about
which field to treat as boolean would have to be inserted, but
I would not know a good place that of the top of my head.

>   3) Yonik is currently looking into what kind of patch it would take to
>      optionally turn off norms (I'm not sure if he's looking at doing it
>      "per field" or "per index").  Is that the kind of thing that would
>      even be considered for getting commited?

For query searching the problem is  the norms in RAM, not on disk.
Treating a field consistently as boolean would avoid reading the
norms from disk.

For really large indexes the norms might become a bottleneck for 
when building them, but iirc this was improved recently.

Regards,
Paul Elschot


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message