lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Mastracci (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (LUCENE-753) Use NIO positional read to avoid synchronization in FSIndexInput
Date Tue, 29 Jul 2008 17:57:32 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617880#action_12617880
] 

mmastrac edited comment on LUCENE-753 at 7/29/08 10:57 AM:
--------------------------------------------------------------------

I just tried out the latest NIOFSDirectory patch and I'm seeing a bug.  If I go back to the
regular FSDirectory, everything works fine.

I can't reproduce it on a smaller testcase.  It only happens with the live index.

Any ideas on where to debug?

Caused by: java.lang.IndexOutOfBoundsException: Index: 24444, Size: 4
	at java.util.ArrayList.RangeCheck(ArrayList.java:547)
	at java.util.ArrayList.get(ArrayList.java:322)
	at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:260)
	at org.apache.lucene.index.FieldInfos.fieldName(FieldInfos.java:249)
	at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:68)
	at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123)
	at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154)
	at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223)
	at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217)
	at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678)
	at org.apache.lucene.index.MultiSegmentReader.docFreq(MultiSegmentReader.java:373)
	at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
	at org.apache.lucene.search.Similarity.idf(Similarity.java:457)
	at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
	at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146)
	at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
	at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
	at org.apache.lucene.search.Query.weight(Query.java:95)
	at org.apache.lucene.search.Searcher.createWeight(Searcher.java:171)
	at org.apache.lucene.search.Searcher.search(Searcher.java:132)

The index is not using the compound file format:
<pre>
7731499698 Jul 28 03:46 _6zk.fdt
 232014520 Jul 28 03:50 _6zk.fdx
        32 Jul 28 03:50 _6zk.fnm
3775713450 Jul 28 04:06 _6zk.frq
  58003634 Jul 28 04:07 _6zk.nrm
2944298834 Jul 28 04:18 _6zk.prx
    432418 Jul 28 04:18 _6zk.tii
  30784106 Jul 28 04:19 _6zk.tis
 217354711 Jul 28 08:18 _76i.fdt
   6509864 Jul 28 08:18 _76i.fdx
        32 Jul 28 08:18 _76i.fnm
 144348761 Jul 28 08:18 _76i.frq
   1627470 Jul 28 08:18 _76i.nrm
 295528445 Jul 28 08:19 _76i.prx
     52622 Jul 28 08:19 _76i.tii
   3858378 Jul 28 08:19 _76i.tis
 199621206 Jul 29 13:29 _7cm.fdt
   5994720 Jul 29 13:29 _7cm.fdx
        32 Jul 29 13:29 _7cm.fnm
 136445620 Jul 29 13:29 _7cm.frq
   1498684 Jul 29 13:29 _7cm.nrm
 284805312 Jul 29 13:30 _7cm.prx
     48346 Jul 29 13:30 _7cm.tii
   3522117 Jul 29 13:30 _7cm.tis
   3914068 Jul 29 13:30 _7cn.fdt
    119184 Jul 29 13:30 _7cn.fdx
        32 Jul 29 13:30 _7cn.fnm
   2993343 Jul 29 13:30 _7cn.frq
     29800 Jul 29 13:30 _7cn.nrm
   7380878 Jul 29 13:30 _7cn.prx
      5277 Jul 29 13:30 _7cn.tii
    378816 Jul 29 13:30 _7cn.tis
    383147 Jul 29 13:30 _7cq.fdt
     11240 Jul 29 13:30 _7cq.fdx
        32 Jul 29 13:30 _7cq.fnm
    290398 Jul 29 13:30 _7cq.frq
      2814 Jul 29 13:30 _7cq.nrm
    763135 Jul 29 13:30 _7cq.prx
      1581 Jul 29 13:30 _7cq.tii
    115971 Jul 29 13:30 _7cq.tis
        19 Jul 29 13:30 date
        20 Jul 21 01:53 segments.gen
       155 Jul 29 13:30 segments_d61
<pre>

      was (Author: mmastrac):
    I just tried out the latest NIOFSDirectory patch and I'm seeing a bug.  If I go back to
the regular FSDirectory, everything works fine.

I can't reproduce it on a smaller testcase.  It only happens with the live index.

Any ideas on where to debug?

Caused by: java.lang.IndexOutOfBoundsException: Index: 24444, Size: 4
	at java.util.ArrayList.RangeCheck(ArrayList.java:547)
	at java.util.ArrayList.get(ArrayList.java:322)
	at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:260)
	at org.apache.lucene.index.FieldInfos.fieldName(FieldInfos.java:249)
	at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:68)
	at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123)
	at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154)
	at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223)
	at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217)
	at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678)
	at org.apache.lucene.index.MultiSegmentReader.docFreq(MultiSegmentReader.java:373)
	at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
	at org.apache.lucene.search.Similarity.idf(Similarity.java:457)
	at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
	at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146)
	at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
	at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
	at org.apache.lucene.search.Query.weight(Query.java:95)
	at org.apache.lucene.search.Searcher.createWeight(Searcher.java:171)
	at org.apache.lucene.search.Searcher.search(Searcher.java:132)

The index is not using the compound file format:
{{
7731499698 Jul 28 03:46 _6zk.fdt
 232014520 Jul 28 03:50 _6zk.fdx
        32 Jul 28 03:50 _6zk.fnm
3775713450 Jul 28 04:06 _6zk.frq
  58003634 Jul 28 04:07 _6zk.nrm
2944298834 Jul 28 04:18 _6zk.prx
    432418 Jul 28 04:18 _6zk.tii
  30784106 Jul 28 04:19 _6zk.tis
 217354711 Jul 28 08:18 _76i.fdt
   6509864 Jul 28 08:18 _76i.fdx
        32 Jul 28 08:18 _76i.fnm
 144348761 Jul 28 08:18 _76i.frq
   1627470 Jul 28 08:18 _76i.nrm
 295528445 Jul 28 08:19 _76i.prx
     52622 Jul 28 08:19 _76i.tii
   3858378 Jul 28 08:19 _76i.tis
 199621206 Jul 29 13:29 _7cm.fdt
   5994720 Jul 29 13:29 _7cm.fdx
        32 Jul 29 13:29 _7cm.fnm
 136445620 Jul 29 13:29 _7cm.frq
   1498684 Jul 29 13:29 _7cm.nrm
 284805312 Jul 29 13:30 _7cm.prx
     48346 Jul 29 13:30 _7cm.tii
   3522117 Jul 29 13:30 _7cm.tis
   3914068 Jul 29 13:30 _7cn.fdt
    119184 Jul 29 13:30 _7cn.fdx
        32 Jul 29 13:30 _7cn.fnm
   2993343 Jul 29 13:30 _7cn.frq
     29800 Jul 29 13:30 _7cn.nrm
   7380878 Jul 29 13:30 _7cn.prx
      5277 Jul 29 13:30 _7cn.tii
    378816 Jul 29 13:30 _7cn.tis
    383147 Jul 29 13:30 _7cq.fdt
     11240 Jul 29 13:30 _7cq.fdx
        32 Jul 29 13:30 _7cq.fnm
    290398 Jul 29 13:30 _7cq.frq
      2814 Jul 29 13:30 _7cq.nrm
    763135 Jul 29 13:30 _7cq.prx
      1581 Jul 29 13:30 _7cq.tii
    115971 Jul 29 13:30 _7cq.tis
        19 Jul 29 13:30 date
        20 Jul 21 01:53 segments.gen
       155 Jul 29 13:30 segments_d61
}}
  
> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
>                 Key: LUCENE-753
>                 URL: https://issues.apache.org/jira/browse/LUCENE-753
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Store
>            Reporter: Yonik Seeley
>         Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java,
FileReadTest.java, FileReadTest.java, FileReadTest.java, FSDirectoryPool.patch, FSIndexInput.patch,
FSIndexInput.patch, lucene-753.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying
file.
> This could mitigate any MT performance drop caused by reducing the number of files in
the index format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message