lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] Updated: (LUCENE-2944) BytesRef reuse bugs in QueryParser and analysis.jsp
Date Wed, 02 Mar 2011 04:24:36 GMT


Robert Muir updated LUCENE-2944:

    Attachment: LUCENE-2944.patch

I think producer should own the BytesRef and if consumer wants to hang onto it, it should
make a deep copy? This is consistent w/ TermAttribute...

Here's a new patch implementing it this way. I refactored TermToBytesRefAttribute into two
methods, getBytesRef() and hash()... I find this less confusing, removes some wasted bytesrefs
being needlessly created here and there (e.g. queryparser), and does things like allow an
attribute to say, pre-size its reusable BytesRef to a huge size or other things custom attributes
might want to do.  

Here is the consumer code sample I added from the javadoc:
 * Consumers of this attribute call getBytesRef() up-front, and then
 * invoke hash() for each term. Example:
final TermToBytesRefAttribute termAtt = tokenStream.getAttribute(TermToBytesRefAttribute.class);
final BytesRef bytes = termAtt.getBytesRef();

while (termAtt.incrementToken() {

   * you must call termAtt.hash() even if you don't need this hashCode.
   * this encodes the term value (internally it might be a char[], etc) into the bytes.
  int hashCode = termAtt.hash();

  if (isInteresting(bytes)) {
     * because the bytes are reused by the attribute (like CharTermAttribute's char[] buffer),
     * you should make a copy if you need persistent access to the bytes, otherwise they will
     * be rewritten across calls to incrementToken()
    doSomethingWith(new BytesRef(bytes));

> BytesRef reuse bugs in QueryParser and analysis.jsp
> ---------------------------------------------------
>                 Key: LUCENE-2944
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>             Fix For: 4.0
>         Attachments: LUCENE-2944.patch, LUCENE-2944.patch, LUCENE-2944.patch, LUCENE-2944_option2.patch
> Some code uses BytesRef as if it were a "String", in this case consumers of TermToBytesRefAttribute.
> The thing is, while our general implementation works on char[] and then populates the
consumers BytesRef,
> not all TermToBytesRefAttribute implementations do this, specifically ICU collation,
it reuses the bytes and simply sets the pointers:
> {noformat}
>   @Override
>   public int toBytesRef(BytesRef target) {
>     collator.getRawCollationKey(toString(), key);
>     target.bytes = key.bytes;
>     target.offset = 0;
>     target.length = key.size;
>     return target.hashCode();
>   }
> {noformat}
> Most of the blame falls on me as I added this to the queryparser in LUCENE-2514.
> Attached is a patch so that these consumers re-use a 'spare' and copy the bytes when
they are going to make a long lasting object such as a Term.

This message is automatically generated by JIRA.
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message