lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nico Krijnen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2729) Index corruption after 'read past EOF' under heavy update load and snapshot export
Date Mon, 01 Nov 2010 08:43:23 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926859#action_12926859
] 

Nico Krijnen commented on LUCENE-2729:
--------------------------------------

A second file listing from another test run, same result: read past EOF

{code}
jteb:assetIndex jteb$ ls -la
total 38739848
drwxr-xr-x  2 jteb  jteb        4964 26 okt 11:51 .
drwxr-xr-x  3 jteb  jteb         204 22 okt 11:42 ..
-rw-r--r--  1 jteb  jteb   969134416 18 okt 16:41 _2q.fdt
-rw-r--r--  1 jteb  jteb       36652 18 okt 16:41 _2q.fdx
-rw-r--r--  1 jteb  jteb         276 18 okt 16:41 _2q.fnm
-rw-r--r--  1 jteb  jteb     4685726 18 okt 16:41 _2q.frq
-rw-r--r--  1 jteb  jteb        9166 18 okt 16:41 _2q.nrm
-rw-r--r--  1 jteb  jteb   393230403 18 okt 16:42 _2q.prx
-rw-r--r--  1 jteb  jteb        7447 18 okt 16:42 _2q.tii
-rw-r--r--  1 jteb  jteb      746299 18 okt 16:42 _2q.tis
-rw-r--r--  1 jteb  jteb        8394 18 okt 16:42 _2q.tvd
-rw-r--r--  1 jteb  jteb   599185081 18 okt 16:42 _2q.tvf
-rw-r--r--  1 jteb  jteb       73300 18 okt 16:42 _2q.tvx
-rw-r--r--  1 jteb  jteb  2061261675 18 okt 16:44 _39.fdt
-rw-r--r--  1 jteb  jteb        1012 18 okt 16:44 _39.fdx
-rw-r--r--  1 jteb  jteb         276 18 okt 16:44 _39.fnm
-rw-r--r--  1 jteb  jteb    17754579 18 okt 16:44 _39.frq
-rw-r--r--  1 jteb  jteb         256 18 okt 16:44 _39.nrm
-rw-r--r--  1 jteb  jteb   121067407 18 okt 16:44 _39.prx
-rw-r--r--  1 jteb  jteb      137511 18 okt 16:44 _39.tii
-rw-r--r--  1 jteb  jteb    11726653 18 okt 16:44 _39.tis
-rw-r--r--  1 jteb  jteb         185 18 okt 16:44 _39.tvd
-rw-r--r--  1 jteb  jteb   233037042 18 okt 16:44 _39.tvf
-rw-r--r--  1 jteb  jteb        2020 18 okt 16:44 _39.tvx
-rw-r--r--  1 jteb  jteb  1595882722 18 okt 16:45 _3u.fdt
-rw-r--r--  1 jteb  jteb       63692 18 okt 16:45 _3u.fdx
-rw-r--r--  1 jteb  jteb         330 18 okt 16:45 _3u.fnm
-rw-r--r--  1 jteb  jteb     8001869 18 okt 16:45 _3u.frq
-rw-r--r--  1 jteb  jteb       15926 18 okt 16:45 _3u.nrm
-rw-r--r--  1 jteb  jteb   647374863 18 okt 16:45 _3u.prx
-rw-r--r--  1 jteb  jteb       11319 18 okt 16:45 _3u.tii
-rw-r--r--  1 jteb  jteb     1168399 18 okt 16:45 _3u.tis
-rw-r--r--  1 jteb  jteb       14209 18 okt 16:45 _3u.tvd
-rw-r--r--  1 jteb  jteb   986370136 18 okt 16:46 _3u.tvf
-rw-r--r--  1 jteb  jteb      127380 18 okt 16:46 _3u.tvx
-rw-r--r--  1 jteb  jteb  2057147455 18 okt 16:47 _3v.fdt
-rw-r--r--  1 jteb  jteb         476 18 okt 16:47 _3v.fdx
-rw-r--r--  1 jteb  jteb         384 18 okt 16:47 _3v.fnm
-rw-r--r--  1 jteb  jteb    15201111 18 okt 16:47 _3v.frq
-rw-r--r--  1 jteb  jteb         122 18 okt 16:47 _3v.nrm
-rw-r--r--  1 jteb  jteb   109724024 18 okt 16:47 _3v.prx
-rw-r--r--  1 jteb  jteb      132491 18 okt 16:47 _3v.tii
-rw-r--r--  1 jteb  jteb    11457688 18 okt 16:47 _3v.tis
-rw-r--r--  1 jteb  jteb         114 18 okt 16:47 _3v.tvd
-rw-r--r--  1 jteb  jteb   211902147 18 okt 16:48 _3v.tvf
-rw-r--r--  1 jteb  jteb         948 18 okt 16:48 _3v.tvx
-rw-r--r--  1 jteb  jteb  2691565961 18 okt 16:49 _4c.fdt
-rw-r--r--  1 jteb  jteb       39572 18 okt 16:49 _4c.fdx
-rw-r--r--  1 jteb  jteb         276 18 okt 16:49 _4c.fnm
-rw-r--r--  1 jteb  jteb    18724620 18 okt 16:49 _4c.frq
-rw-r--r--  1 jteb  jteb        9896 18 okt 16:49 _4c.nrm
-rw-r--r--  1 jteb  jteb   590255960 18 okt 16:50 _4c.prx
-rw-r--r--  1 jteb  jteb      141243 18 okt 16:50 _4c.tii
-rw-r--r--  1 jteb  jteb    12185869 18 okt 16:50 _4c.tis
-rw-r--r--  1 jteb  jteb        9894 18 okt 16:50 _4c.tvd
-rw-r--r--  1 jteb  jteb   932649779 18 okt 16:51 _4c.tvf
-rw-r--r--  1 jteb  jteb       79140 18 okt 16:51 _4c.tvx
-rw-r--r--  1 jteb  jteb  2398908136 18 okt 16:52 _4d.fdt
-rw-r--r--  1 jteb  jteb         548 18 okt 16:52 _4d.fdx
-rw-r--r--  1 jteb  jteb         354 18 okt 16:52 _4d.fnm
-rw-r--r--  1 jteb  jteb    24581614 18 okt 16:52 _4d.frq
-rw-r--r--  1 jteb  jteb         140 18 okt 16:52 _4d.nrm
-rw-r--r--  1 jteb  jteb   158243133 18 okt 16:52 _4d.prx
-rw-r--r--  1 jteb  jteb      141948 18 okt 16:52 _4d.tii
-rw-r--r--  1 jteb  jteb    12259425 18 okt 16:52 _4d.tis
-rw-r--r--  1 jteb  jteb         140 18 okt 16:52 _4d.tvd
-rw-r--r--  1 jteb  jteb   303769970 18 okt 16:53 _4d.tvf
-rw-r--r--  1 jteb  jteb        1092 18 okt 16:53 _4d.tvx
-rw-r--r--  1 jteb  jteb  1081212027 18 okt 16:53 _4p.fdt
-rw-r--r--  1 jteb  jteb         212 18 okt 16:53 _4p.fdx
-rw-r--r--  1 jteb  jteb         354 18 okt 16:53 _4p.fnm
-rw-r--r--  1 jteb  jteb     8294102 18 okt 16:53 _4p.frq
-rw-r--r--  1 jteb  jteb          56 18 okt 16:53 _4p.nrm
-rw-r--r--  1 jteb  jteb    60513257 18 okt 16:53 _4p.prx
-rw-r--r--  1 jteb  jteb      134898 18 okt 16:53 _4p.tii
-rw-r--r--  1 jteb  jteb    11376730 18 okt 16:53 _4p.tis
-rw-r--r--  1 jteb  jteb          56 18 okt 16:53 _4p.tvd
-rw-r--r--  1 jteb  jteb   116715012 18 okt 16:53 _4p.tvf
-rw-r--r--  1 jteb  jteb         420 18 okt 16:53 _4p.tvx
-rw-r--r--  1 jteb  jteb   787581180 18 okt 16:54 _55.fdt
-rw-r--r--  1 jteb  jteb         156 18 okt 16:54 _55.fdx
-rw-r--r--  1 jteb  jteb         354 18 okt 16:54 _55.fnm
-rw-r--r--  1 jteb  jteb     5376741 18 okt 16:54 _55.frq
-rw-r--r--  1 jteb  jteb          42 18 okt 16:54 _55.nrm
-rw-r--r--  1 jteb  jteb    39539985 18 okt 16:54 _55.prx
-rw-r--r--  1 jteb  jteb      133483 18 okt 16:54 _55.tii
-rw-r--r--  1 jteb  jteb    11351989 18 okt 16:54 _55.tis
-rw-r--r--  1 jteb  jteb          42 18 okt 16:54 _55.tvd
-rw-r--r--  1 jteb  jteb    75958725 18 okt 16:54 _55.tvf
-rw-r--r--  1 jteb  jteb         308 18 okt 16:54 _55.tvx
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6i.fdt
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6i.fdx
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6i.frq
-rw-r--r--  1 jteb  jteb           0 22 okt 12:05 _6i.nrm
-rw-r--r--  1 jteb  jteb           0 22 okt 12:12 _6i.tvd
-rw-r--r--  1 jteb  jteb           0 22 okt 12:12 _6i.tvf
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6i.tvx
-rw-r--r--  1 jteb  jteb           0 22 okt 12:07 _6j.nrm
-rw-r--r--  1 jteb  jteb           0 22 okt 12:08 _6k.fdt
-rw-r--r--  1 jteb  jteb           0 22 okt 12:08 _6k.fdx
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6k.fnm
-rw-r--r--  1 jteb  jteb           0 22 okt 12:05 _6k.tis
-rw-r--r--  1 jteb  jteb           0 22 okt 12:08 _6l.frq
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6l.nrm
-rw-r--r--  1 jteb  jteb           0 22 okt 12:11 _6l.prx
-rw-r--r--  1 jteb  jteb   356417086 22 okt 12:02 _6u.fdt
-rw-r--r--  1 jteb  jteb      172180 22 okt 12:02 _6u.fdx
-rw-r--r--  1 jteb  jteb         723 22 okt 12:02 _6u.fnm
-rw-r--r--  1 jteb  jteb     8585718 22 okt 12:02 _6u.frq
-rw-r--r--  1 jteb  jteb       43048 22 okt 12:02 _6u.nrm
-rw-r--r--  1 jteb  jteb    18540155 22 okt 12:02 _6u.prx
-rw-r--r--  1 jteb  jteb      185484 22 okt 12:02 _6u.tii
-rw-r--r--  1 jteb  jteb    15851802 22 okt 12:02 _6u.tis
-rw-r--r--  1 jteb  jteb       21534 22 okt 12:02 _6u.tvd
-rw-r--r--  1 jteb  jteb    34227796 22 okt 12:02 _6u.tvf
-rw-r--r--  1 jteb  jteb      344356 22 okt 12:02 _6u.tvx
-rw-r--r--  1 jteb  jteb          39 22 okt 12:05 _6u_1.del
-rw-r--r--  1 jteb  jteb      316687 22 okt 12:05 _6v.fdt
-rw-r--r--  1 jteb  jteb        2716 22 okt 12:05 _6v.fdx
-rw-r--r--  1 jteb  jteb         593 22 okt 12:05 _6v.fnm
-rw-r--r--  1 jteb  jteb       93284 22 okt 12:05 _6v.frq
-rw-r--r--  1 jteb  jteb         343 22 okt 12:05 _6v.nrm
-rw-r--r--  1 jteb  jteb       12416 22 okt 12:05 _6v.prx
-rw-r--r--  1 jteb  jteb        1679 22 okt 12:05 _6v.tii
-rw-r--r--  1 jteb  jteb      120258 22 okt 12:05 _6v.tis
-rw-r--r--  1 jteb  jteb      319766 22 okt 12:05 _6w.fdt
-rw-r--r--  1 jteb  jteb        2756 22 okt 12:05 _6w.fdx
-rw-r--r--  1 jteb  jteb         593 22 okt 12:05 _6w.fnm
-rw-r--r--  1 jteb  jteb       94316 22 okt 12:05 _6w.frq
-rw-r--r--  1 jteb  jteb         348 22 okt 12:05 _6w.nrm
-rw-r--r--  1 jteb  jteb       12593 22 okt 12:05 _6w.prx
-rw-r--r--  1 jteb  jteb        1695 22 okt 12:05 _6w.tii
-rw-r--r--  1 jteb  jteb      117984 22 okt 12:05 _6w.tis
-rw-r--r--  1 jteb  jteb      300004 22 okt 12:05 _6x.fdt
-rw-r--r--  1 jteb  jteb        2548 22 okt 12:05 _6x.fdx
-rw-r--r--  1 jteb  jteb         593 22 okt 12:05 _6x.fnm
-rw-r--r--  1 jteb  jteb       86935 22 okt 12:05 _6x.frq
-rw-r--r--  1 jteb  jteb         322 22 okt 12:05 _6x.nrm
-rw-r--r--  1 jteb  jteb       11811 22 okt 12:05 _6x.prx
-rw-r--r--  1 jteb  jteb        1578 22 okt 12:05 _6x.tii
-rw-r--r--  1 jteb  jteb      111238 22 okt 12:05 _6x.tis
-rw-r--r--  1 jteb  jteb         216 22 okt 13:24 index.diagnostic
-rw-r--r--  1 jteb  jteb           5 22 okt 12:05 index.directory
-rw-r--r--  1 jteb  jteb          20 22 okt 12:05 segments.gen
-rw-r--r--  1 jteb  jteb           0 22 okt 12:05 segments_36
-rw-r--r--  1 jteb  jteb        2877 22 okt 12:05 segments_3g
{code}

> Index corruption after 'read past EOF' under heavy update load and snapshot export
> ----------------------------------------------------------------------------------
>
>                 Key: LUCENE-2729
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2729
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 3.0.1, 3.0.2
>         Environment: Happens on both OS X 10.6 and Windows 2008 Server. Integrated with
zoie (using a zoie snapshot from 2010-08-06: zoie-2.0.0-snapshot-20100806.jar).
>            Reporter: Nico Krijnen
>
> We have a system running lucene and zoie. We use lucene as a content store for a CMS/DAM
system. We use the hot-backup feature of zoie to make scheduled backups of the index. This
works fine for small indexes and when there are not a lot of changes to the index when the
backup is made.
> On large indexes (about 5 GB to 19 GB), when a backup is made while the index is being
changed a lot (lots of document additions and/or deletions), we almost always get a 'read
past EOF' at some point, followed by lots of 'Lock obtain timed out'.
> At that point we get lots of 0 kb files in the index, data gets lots, and the index is
unusable.
> When we stop our server, remove the 0kb files and restart our server, the index is operational
again, but data has been lost.
> I'm not sure if this is a zoie or a lucene issue, so i'm posting it to both. Hopefully
someone has some ideas where to look to fix this.
> Some more details...
> Stack trace of the read past EOF and following Lock obtain timed out:
> {code}
> 78307 [proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader@31ca5085] ERROR proj.zoie.impl.indexing.internal.BaseSearchIndex
- read past EOF
> java.io.IOException: read past EOF
>     at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:154)
>     at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
>     at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:37)
>     at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:69)
>     at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:245)
>     at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:166)
>     at org.apache.lucene.index.DirectoryReader.doCommit(DirectoryReader.java:725)
>     at org.apache.lucene.index.IndexReader.commit(IndexReader.java:987)
>     at org.apache.lucene.index.IndexReader.commit(IndexReader.java:973)
>     at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:162)
>     at org.apache.lucene.index.IndexReader.close(IndexReader.java:1003)
>     at proj.zoie.impl.indexing.internal.BaseSearchIndex.deleteDocs(BaseSearchIndex.java:203)
>     at proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:223)
>     at proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153)
>     at proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134)
>     at proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171)
>     at proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373)
> 579336 [proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader@31ca5085] ERROR proj.zoie.impl.indexing.internal.LuceneIndexDataLoader
- Problem copying segments: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@5ad0b895:
write.lock
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@5ad0b895:
write.lock
>     at org.apache.lucene.store.Lock.obtain(Lock.java:84)
>     at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1060)
>     at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:957)
>     at proj.zoie.impl.indexing.internal.DiskSearchIndex.openIndexWriter(DiskSearchIndex.java:176)
>     at proj.zoie.impl.indexing.internal.BaseSearchIndex.loadFromIndex(BaseSearchIndex.java:228)
>     at proj.zoie.impl.indexing.internal.LuceneIndexDataLoader.loadFromIndex(LuceneIndexDataLoader.java:153)
>     at proj.zoie.impl.indexing.internal.DiskLuceneIndexDataLoader.loadFromIndex(DiskLuceneIndexDataLoader.java:134)
>     at proj.zoie.impl.indexing.internal.RealtimeIndexDataLoader.processBatch(RealtimeIndexDataLoader.java:171)
>     at proj.zoie.impl.indexing.internal.BatchedIndexDataLoader$LoaderThread.run(BatchedIndexDataLoader.java:373)
> {code}
> We get exactly the same behavour on both OS X and on Windows. On both zoie is using a
SimpleFSDirectory.
> We also use a SingleInstanceLockFactory (since our process is the only one working with
the index), but we get the same behaviour with a NativeFSLock.
> The snapshot backup is being made by calling:
> *proj.zoie.impl.indexing.ZoieSystem.exportSnapshot(WritableByteChannel)*
> Same issue in zoie JIRA:
> http://snaprojects.jira.com/browse/ZOIE-51

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message