Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Fri, 3 Jun 2011 21:53:48 +0000 (UTC)
From: "KuroSaka TeruHiko (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: 
 <394621775.67346.1307138028097.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Issue Comment Edited] (LUCENE-2645) False assertion of >0
 position delta in StandardPostingsWriterImpl
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044077#comment-13044077 ] 

KuroSaka TeruHiko edited comment on LUCENE-2645 at 6/3/11 9:53 PM:
-------------------------------------------------------------------

This is preventing one of unit tests for our Tokenizer/Filter product to fail.  Although it looks as if the unit test case were failing, a closer look shows that the failure is from this this assertion inside the Lucene core.  
We have inspected the Token sequence that causes this assertion to fail, and it looks valid.
The attached program can reproduce this false assertion failure. Please give this bug a higher priority, and remove this assertion.
{noformat}
$ javac -cp lucene-core-4.0-SNAPSHOT.jar LuceneTrunkAssertErrorReproducer.java
$ java -ea -cp .:lucene-core-4.0-SNAPSHOT.jar LuceneTrunkAssertErrorReproducer
Exception in thread "main" java.lang.AssertionError: position=1 lastPosition=1
	at org.apache.lucene.index.codecs.standard.StandardPostingsWriter.addPosition(StandardPostingsWriter.java:197)
	at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:385)
	at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:88)
	at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
	at org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
	at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:75)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:457)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:421)
	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:548)
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2776)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2753)
	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2619)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2692)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2674)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2658)
	at LuceneTrunkAssertErrorReproducer.main(LuceneTrunkAssertErrorReproducer.java:39)
{noformat}

      was (Author: tkurosaka):
    This is preventing one of unit tests for our Tokenizer/Filter product to fail.  Although it looks as though the unit test is failing, actually it is an assertion error from this assertion.  
We have analyzed the Token sequence that causes this assertion to fail, but they look valid.
The attached program can reproduce this false assertion failure. Please give it higher priority and remove this assertion.
{noformat}
$ javac -cp lucene-core-4.0-SNAPSHOT.jar LuceneTrunkAssertErrorReproducer.java
$ java -ea -cp .:lucene-core-4.0-SNAPSHOT.jar LuceneTrunkAssertErrorReproducer
Exception in thread "main" java.lang.AssertionError: position=1 lastPosition=1
	at org.apache.lucene.index.codecs.standard.StandardPostingsWriter.addPosition(StandardPostingsWriter.java:197)
	at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:385)
	at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:88)
	at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
	at org.apache.lucene.index.DocInverter.flush(DocInverter.java:80)
	at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:75)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:457)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:421)
	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:548)
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:2776)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2753)
	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2619)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2692)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2674)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2658)
	at LuceneTrunkAssertErrorReproducer.main(LuceneTrunkAssertErrorReproducer.java:39)
{noformat}
  
> False assertion of >0 position delta in StandardPostingsWriterImpl
> ------------------------------------------------------------------
>
>                 Key: LUCENE-2645
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2645
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: core/index
>            Reporter: David Smiley
>            Priority: Minor
>         Attachments: LuceneTrunkAssertErrorReproducer.java
>
>
> StandardPostingsWriterImpl line 159 is:
> {code:java}
>     assert delta > 0 || position == 0 || position == -1: "position=" + position + " lastPosition=" + lastPosition;            // not quite right (if pos=0 is repeated twice we don't catch it)
> {code}
> I enable assertions when I run my unit tests and I've found this assertion to fail when delta is 0 which occurs when the same position value is sent in twice in arrow.  Once I added RemoveDuplicatesTokenFilter, this problem went away.  Should I really be forced to add this filter?  I think delta >= 0 would be a better assertion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org