lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] [Commented] (LUCENE-6504) implement norms with random access API
Date Wed, 27 May 2015 14:13:17 GMT


Robert Muir commented on LUCENE-6504:

Before/now, AFAIK norms take 1 byte per field per doc of heap. I looked over the patch briefly;
does this essentially put norms off-heap?

In the worst case. Currently they are compressed with bitpacking and other tricks to try to
be reasonable. But what was missing all along was a random access api in Directory so that
this can just be MappedByteBuffer.get(long) (see linked issue and justification). If you want
them to be in heap memory, use fileswitchdirectory and ramdirectory.

Does this patch also bring in accurate norms or is something else required to enable actually
utilize that?

You have 1 byte norms because your chosen similarity squashes to that, but the interface between
similarity and indexwriter is "long" since lucene 4 and all codecs test and support that.

> implement norms with random access API
> --------------------------------------
>                 Key: LUCENE-6504
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Robert Muir
>         Attachments: LUCENE-6504.patch
> We added this api in LUCENE-5729 but we never explored implementing norms with it. These
are generally the largest consumer of heap memory and often a real hassle for users.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message