lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barakat Barakat (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4583) StraightBytesDocValuesField fails if bytes > 32k
Date Mon, 03 Dec 2012 21:03:58 GMT


Barakat Barakat commented on LUCENE-4583:

The limitation comes from PagedBytes. When PagedBytes is created it is given a number of bits
to use per block. The blockSize is set to (1 << blockBits). From what I've seen, classes
that use PagedBytes usually pass in 15 as the blockBits. This leads to the 32768 byte limit.

The fillSlice function of the PagedBytes.Reader will return a block of bytes that is either
inside one block or overlapping two blocks. If you try to give it a length that is over the
block size it will hit the out of bounds exception. For the project I am working on, we need
more than 32k bytes for our DocValues. We need that much rarely, but we still need that much
to keep the search functioning. I fixed this for our project by changing fillSlices to this:

Test unit:

After placing this in our Solr instance, the search no longer crashes and returns the correct
values when the document has a DocValues field more than 32k bytes. As far as I know there
is no limit now. I haven't noticed a performance hit. It shouldn't really affect performance
unless you have many of these large DocValues fields. Thank you to David for his help with
> StraightBytesDocValuesField fails if bytes > 32k
> ------------------------------------------------
>                 Key: LUCENE-4583
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/index
>    Affects Versions: 4.0, 4.1, 5.0
>            Reporter: David Smiley
>            Priority: Critical
> I didn't observe any limitations on the size of a bytes based DocValues field value in
the docs.  It appears that the limit is 32k, although I didn't get any friendly error telling
me that was the limit.  32k is kind of small IMO; I suspect this limit is unintended and as
such is a bug.    The following test fails:
> {code:java}
>   public void testBigDocValue() throws IOException {
>     Directory dir = newDirectory();
>     IndexWriter writer = new IndexWriter(dir, writerConfig(false));
>     Document doc = new Document();
>     BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
>     bytes.length = bytes.bytes.length;//byte data doesn't matter
>     doc.add(new StraightBytesDocValuesField("dvField", bytes));
>     writer.addDocument(doc);
>     writer.commit();
>     writer.close();
>     DirectoryReader reader =;
>     DocValues docValues = MultiDocValues.getDocValues(reader, "dvField");
>     docValues.getSource().getBytes(0, bytes);
>     reader.close();
>     dir.close();
>   }
> {code}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message