Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Message-ID: <5022131.146661280873896398.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 18:18:16 -0400 (EDT)
From: "Sanne Grinovero (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-2585) DirectoryReader.isCurrent might
 fail to see the segments file during concurrent index changes
In-Reply-To: <24868225.144651280867355984.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/LUCENE-2585?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
95059#action_12895059 ]=20

Sanne Grinovero commented on LUCENE-2585:
-----------------------------------------

sure, the test is totally open source; the directory implementation based o=
n Infinispan is hosted as submodule of Infinispan:
http://anonsvn.jboss.org/repos/infinispan/branches/4.1.x/lucene-directory/

The test is
org.infinispan.lucene.profiling.PerformanceCompareStressTest

it is included in the default test suite but disabled in Maven's configurat=
ion, so you should run it manually
mvn clean test -Dtest=3DPerformanceCompareStressTest
(running it requires the jboss.org repositories to be enabled in maven sett=
ings)

To describe it at higher level: there are 5 IndexRead-ing threads using reo=
pen() before each search, 2 threads writing to the index, 1 additional thre=
ad as a coordinator and asserting that readers find what they expect to see=
 in the index.
Exactly the same test scenario is then applied in sequence to RAMDirectory =
(not having issues), NIOFSDirectory, and 4 differently configured Infinispa=
n directories.
Only the FSDirectory is affected by the issue, and it can never complete th=
e full hour of stresstest succesfully, while all other implementations beha=
ve fine.

IndexWriter is set to MaxMergeDocs(5000) and setUseCompoundFile(false); the=
 issue is reveled both using SerialMergeScheduler and while using the defau=
lt merger.

During the last execution the test managed to perform 22,192,006 searches a=
nd 26,875 writes during the hour, but the "benchmark" is invalidated as one=
 thread was killed by the exception.

If you deem it useful I'd be happy in contributing a similar testcase to Lu=
cene, but I assume you won't be excited in having such a long running test.=
 Open to ideas to build a simpler one.


> DirectoryReader.isCurrent might fail to see the segments file during conc=
urrent index changes
> -------------------------------------------------------------------------=
--------------------
>
>                 Key: LUCENE-2585
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2585
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.9.3, 3.0, 3.0.1, 3.0.2
>            Reporter: Sanne Grinovero
>             Fix For: 2.9.4, 3.0.3, 3.1
>
>
> I could reproduce the issue several times but only by running long and st=
ressfull benchmarks, the high number of files is likely part of the scenari=
o.
> All tests run on local disk, using ext3.
> Sample stacktrace:
> {noformat}java.io.FileNotFoundException: no segments* file found in org.a=
pache.lucene.store.NIOFSDirectory@/home/sanne/infinispan-41/lucene-director=
y/tempIndexName: files: _2l3.frq _uz.fdt _1q4.fnm _1q0.fdx _4bc.fdt _v2.tis=
 _4ll.fdx _2l8.tii _ux.fnm _3g7.fdx _4bb.tii _4bj.prx _uy.fdx _3g7.prx _2l7=
.frq _2la.fdt _3ge.nrm _2l6.prx _1py.fdx _3g6.nrm _v0.prx _4bi.tii _2l2.tis=
 _v2.fdx _2l3.nrm _2l8.fnm _4bg.tis _2la.tis _uu.fdx _3g6.fdx _1q3.frq _2la=
.frq _4bb.tis _3gb.tii _1pz.tis _2lb.nrm _4lm.nrm _3g9.tii _v0.fdt _2l5.fnm=
 _v2.prx _4ll.tii _4bd.nrm _2l7.fnm _2l4.nrm _1q2.tis _3gb.fdx _4bh.fdx _1p=
z.nrm _ux.fdx _ux.tii _1q6.nrm _3gf.fdx _4lk.fdt _3gd.nrm _v3.fnm _3g8.prx =
_1q2.nrm _4bh.prx _1q0.frq _ux.fdt _1q7.fdt _4bb.fnm _4bf.nrm _4bc.nrm _3gb=
.fdt _4bh.fnm _2l5.tis _1pz.fnm _1py.fnm _3gc.fnm _2l2.prx _2l4.frq _3gc.fd=
t _ux.tis _1q3.prx _2l7.fdx _4bj.nrm _4bj.fdx _4bi.tis _3g9.prx _1q4.prx _v=
3.fdt _1q3.fdx _2l9.fdt _4bh.tis _3gb.nrm _v2.nrm _3gd.tii _2l7.nrm _2lb.ti=
i _4lm.tis _3ga.fdx _1pz.fdt _3g7.fnm _2l3.fnm _4lk.fnm _uz.fnm _2l2.frq _4=
bd.fdx _1q2.fdt _3g7.tis _4bi.frq _4bj.frq _2l7.prx _ux.prx _3gd.fnm _1q4.f=
dt _1q1.fdt _v1.fnm _1py.nrm _3gf.nrm _4be.fdt _1q3.tii _1q1.prx _2l3.fdt _=
4lk.frq _2l4.fdx _4bd.fnm _uw.frq _3g8.fdx _2l6.tii _1q5.frq _1q5.tis _3g8.=
nrm _uw.nrm _v0.tii _v2.fdt _2l7.fdt _v0.tis _uy.tii _3ge.tii _v1.tii _3gb.=
tis _4lm.fdx _4bc.fnm _2lb.frq _2l6.fnm _3g6.tii _3ge.prx _uu.frq _1pz.fdx =
_1q2.fnm _4bi.prx _3gc.frq _2l9.tis _3ge.fdt _uy.fdt _4ll.fnm _3gc.prx _1q7=
.tii _2l5.nrm _uy.nrm _uv.frq _1q6.frq _4ba.tis _3g9.tis _4be.nrm _4bi.fnm =
_ux.frq _1q1.fnm _v0.fnm _2l4.fnm _4ba.fnm _4be.tis _uz.prx _1q6.fdx _uw.ti=
i _2l6.nrm _1pz.prx _2l7.tis _1q7.fdx _2l9.tii _4lk.tii _uz.frq _3g8.frq _4=
bb.prx _1q5.tii _1q5.prx _v2.frq _4bc.tii _1q7.prx _v2.tii _2lb.tis _4bi.fd=
t _uv.nrm _2l2.fnm _4bd.tii _1q7.tis _4bg.fnm _3ga.frq _uu.fnm _2l9.fnm _3g=
a.fnm _uw.fnm _1pz.frq _1q1.fdx _3ge.fdx _2l3.prx _3ga.nrm _uv.fdt _4bb.nrm=
 _1q7.fnm _uv.tis _3gb.fnm _2l6.tis _1pz.tii _uy.fnm _3gf.fdt _3gc.nrm _4bf=
.tis _1q5.fnm _uu.tis _4bh.tii _2l5.fdt _1q6.tii _4bc.tis _3gc.tii _3g9.fnm=
 _2l6.fdt _4bj.fnm _uu.tii _v3.frq _3g9.fdx _v0.nrm _2l7.tii _1q0.fdt _3ge.=
fnm _4bf.fdt _1q6.prx _uz.nrm _4bi.fdx _3gf.fnm _4lm.frq _v0.fdx _4ba.fdt _=
1py.tii _4bf.tii _uw.fdx _2l5.frq _3g9.nrm _v1.fdt _uw.fdt _4bd.frq _4bg.pr=
x _3gd.tis _1q4.tis _2l9.nrm _2la.nrm _v3.tii _4bf.prx _1q1.nrm _4ba.tii _3=
gd.fdx _1q4.tii _4lm.tii _3ga.tis _4bf.fnm write.lock _2l8.prx _2l8.fdt seg=
ments.gen _2lb.fnm _2l4.fdt _1q2.prx _4be.fnm _3gf.prx _2l6.fdx _3g6.fnm _4=
bb.fdt _4bd.tis _4lk.nrm _2l5.fdx _2la.tii _4bd.prx _4ln.fnm _3gf.tis _4ba.=
nrm _v3.prx _uv.prx _1q3.fnm _3ga.tii _uz.tii _3g9.frq _v0.frq _3ge.tis _3g=
6.tis _4ln.prx _3g7.tii _3g8.fdt _3g7.nrm _3ga.prx _2l2.fdx _2l8.fdx _4ba.p=
rx _1py.frq _uz.fdx _2l3.tii _3g6.prx _v3.fdx _1q6.fdt _v1.nrm _2l2.tii _1q=
0.tis _4ba.fdx _4be.tii _4ba.frq _4ll.fdt _4bh.nrm _4lm.fdt _1q7.frq _4lk.t=
is _4bc.frq _1q6.fnm _3g7.frq _uw.tis _3g8.tis _2l9.fdx _2l4.tii _1q4.fdx _=
4be.prx _1q3.nrm _1q0.tii _1q0.fnm _v3.nrm _1py.tis _3g9.fdt _4bh.fdt _4ll.=
nrm _4lk.prx _3gd.prx _1q3.tis _1q2.tii _2l2.nrm _3gd.fdt _2l3.fdx _3g6.fdt=
 _3gd.frq _1q1.tis _4bb.fdx _1q2.frq _1q3.fdt _v1.tis _2l8.frq _3gc.fdx _1q=
1.frq _4bg.frq _4bb.frq _2la.fdx _2l9.frq _uy.tis _uy.prx _4bg.fdx _3gb.prx=
 _uy.frq _1q2.fdx _4lm.prx _2la.prx _2l4.prx _4bg.fdt _4be.frq _1q7.nrm _2l=
5.prx _4bf.frq _v1.prx _4bd.fdt _2l9.prx _1q6.tis _3g8.fnm _4ln.tis _2l3.ti=
s _4bc.fdx _2lb.prx _3gb.frq _3gf.frq _2la.fnm _3ga.fdt _uz.tis _4bg.nrm _u=
v.tii _4bg.tii _3g8.tii _4ll.frq _uv.fnm _2l8.tis _2l8.nrm _2l2.fdt _4bj.ti=
s _4lk.fdx _uw.prx _4bc.prx _4bj.fdt _4be.fdx _1q4.frq _uu.fdt _1q1.tii _2l=
5.tii _2lb.fdt _4bh.frq _3ge.frq _1py.prx _1q5.nrm _v1.fdx _3g7.fdt _4ln.fd=
t _1q4.nrm _1py.fdt _3gc.tis _4ll.prx _v3.tis _4bf.fdx _1q5.fdx _1q0.prx _4=
bi.nrm _4ll.tis _2l4.tis _3gf.tii _v2.fnm _uu.nrm _1q0.nrm _4lm.fnm _uu.prx=
 _2l6.frq _4ln.nrm _ux.nrm _3g6.frq _1q5.fdt _4bj.tii _2lb.fdx _uv.fdx _v1.=
frq
>         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Segm=
entInfos.java:634)
>         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Segm=
entInfos.java:517)
>         at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:30=
6)
>         at org.apache.lucene.index.SegmentInfos.readCurrentVersion(Segmen=
tInfos.java:408)
>         at org.apache.lucene.index.DirectoryReader.isCurrent(DirectoryRea=
der.java:797)
>         at org.apache.lucene.index.DirectoryReader.doReopenNoWriter(Direc=
toryReader.java:407)
>         at org.apache.lucene.index.DirectoryReader.doReopen(DirectoryRead=
er.java:386)
>         at org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader=
.java:348)
>         at org.infinispan.lucene.profiling.LuceneReaderThread.refreshInde=
xReader(LuceneReaderThread.java:79)
>         at org.infinispan.lucene.profiling.LuceneReaderThread.testLoop(Lu=
ceneReaderThread.java:60)
>         at org.infinispan.lucene.profiling.LuceneUserThread.run(LuceneUse=
rThread.java:60)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadP=
oolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {noformat}

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org