lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-2944) BytesRef reuse bugs in QueryParser and analysis.jsp
Date Wed, 02 Mar 2011 04:24:36 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Muir updated LUCENE-2944:
--------------------------------

    Attachment: LUCENE-2944.patch

{quote}
I think producer should own the BytesRef and if consumer wants to hang onto it, it should
make a deep copy? This is consistent w/ TermAttribute...
{quote}

Here's a new patch implementing it this way. I refactored TermToBytesRefAttribute into two
methods, getBytesRef() and hash()... I find this less confusing, removes some wasted bytesrefs
being needlessly created here and there (e.g. queryparser), and does things like allow an
attribute to say, pre-size its reusable BytesRef to a huge size or other things custom attributes
might want to do.  

Here is the consumer code sample I added from the javadoc:
{code}
/* 
 * Consumers of this attribute call getBytesRef() up-front, and then
 * invoke hash() for each term. Example:
 */
final TermToBytesRefAttribute termAtt = tokenStream.getAttribute(TermToBytesRefAttribute.class);
final BytesRef bytes = termAtt.getBytesRef();

while (termAtt.incrementToken() {

  /* 
   * you must call termAtt.hash() even if you don't need this hashCode.
   * this encodes the term value (internally it might be a char[], etc) into the bytes.
   */
  int hashCode = termAtt.hash();

  if (isInteresting(bytes)) {
     
    /* 
     * because the bytes are reused by the attribute (like CharTermAttribute's char[] buffer),
     * you should make a copy if you need persistent access to the bytes, otherwise they will
     * be rewritten across calls to incrementToken()
     */
    doSomethingWith(new BytesRef(bytes));
  }
}
{code}

> BytesRef reuse bugs in QueryParser and analysis.jsp
> ---------------------------------------------------
>
>                 Key: LUCENE-2944
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2944
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>             Fix For: 4.0
>
>         Attachments: LUCENE-2944.patch, LUCENE-2944.patch, LUCENE-2944.patch, LUCENE-2944_option2.patch
>
>
> Some code uses BytesRef as if it were a "String", in this case consumers of TermToBytesRefAttribute.
> The thing is, while our general implementation works on char[] and then populates the
consumers BytesRef,
> not all TermToBytesRefAttribute implementations do this, specifically ICU collation,
it reuses the bytes and simply sets the pointers:
> {noformat}
>   @Override
>   public int toBytesRef(BytesRef target) {
>     collator.getRawCollationKey(toString(), key);
>     target.bytes = key.bytes;
>     target.offset = 0;
>     target.length = key.size;
>     return target.hashCode();
>   }
> {noformat}
> Most of the blame falls on me as I added this to the queryparser in LUCENE-2514.
> Attached is a patch so that these consumers re-use a 'spare' and copy the bytes when
they are going to make a long lasting object such as a Term.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message