lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1192) FileNotFound exception on adding a RAMDirectory to an IndexWriter
Date Tue, 26 Feb 2008 12:23:52 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572449#action_12572449
] 

Michael McCandless commented on LUCENE-1192:
--------------------------------------------


This actually isn't quite a valid use of Lucene's APIs, because the
ramIndexWriter is actively changing the files in the RAMDirectory.

This worked in past versions of Lucene because merges were done
synchronously.  Now, with background merges by default, a merge can
complete and commit changes to the RAMDirectory while
addIndexesNoOptimize is running.  The addIndexesNoOptimize method
requires a "static" Directory, where no writer is actively making
changes.

There are a few ways you could get this to work:

  * Sync all concurrent merges before calling addIndexesNoOptimize, by
    calling ((ConcurrentMergeScheduler) ramIndexWriter.getMergeScheduler()).sync()
 
  * Switch back to SerialMergeScheduler

  * Close the ramIndexWriter before passing its directory to
    addIndexesNoOptimize.  In this case you need to separately hold
    the underlying RAMDirectory since you can't call
    IndexWriter.getDirectory after close.



> FileNotFound exception on adding a RAMDirectory to an IndexWriter
> -----------------------------------------------------------------
>
>                 Key: LUCENE-1192
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1192
>             Project: Lucene - Java
>          Issue Type: Bug
>    Affects Versions: 2.3.1
>         Environment: || Description || Value ||
> | java -version | {noformat}java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
> {noformat} |
> | JVM settings | {noformat}-server -verbose:gc -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime

> -XX:NewRatio=2 -XX:ParallelGCThreads=4 -Xms6000m -Xmx6000m{noformat} |
> | uname -a | {noformat}Linux pdbindexer2 2.6.22.14-72.fc6 #1 
> SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux{noformat} |
> | CPU | Dual Core AMD Opteron(tm) Processor 280 |
> | Total available memory | 8G |
>            Reporter: Alex Falca
>
> Hi,
> I'm getting FileNotFoundException on adding RAMDirectory to an IndexWriter with following
stack trace:
> {noformat}
> java.io.FileNotFoundException: _8.fnm
>         at org.apache.lucene.store.RAMDirectory.openInput(RAMDirectory.java:234)
>         at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:57)
>         at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:298)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>         at org.apache.lucene.index.IndexWriter.copyExternalSegments(IndexWriter.java:2263)
>         at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2238)
> {noformat}
> Same code works fine with Lucene 2.2.0. Basically I'm trying to index a huge number of
documents (~21mln) having following workflow:
> 1. First documents are added to and index which use a RAMDirectory as a storage
> 2. Once a given condition occurs RAM based index writer is flushed and his directory
is being added to another index writer, which use a FSDirectory as a storage. 
> Sample code:
> {code}
> IndexWriter mainIndexWriter = new IndexWriter(indexDir, new JapaneseAnalyzer(), true);
> IndexWriter ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(),
true);
> mainIndexWriter.setMergeFactor(10);
> mainIndexWriter.setUseCompoundFile(false);
> ramIndexWriter.setUseCompoundFile(false);
> // Here some code for adding documents to RAM index writer
> ...................................................................................................................
> // method where I'm flushing from memory to disk and getting exception
> ramIndexWriter.flush();
> mainIndexWriter.addIndexesNoOptimize(new Directory[] {ramIndexWriter.getDirectory()});
> ramIndexWriter.close();
> ramIndexWriter = new IndexWriter(new RAMDirectory(), new JapaneseAnalyzer(), true);
> ramIndexWriter.setUseCompoundFile(false);
> System.gc();
> {code}
> I understand that I should be using new features of IndexWriter like flushing by RAM
Usage, but wanna see if there any bugs in my case. Possible I'm facing issue described in
LUCENE-1175, but need a confirmation on that. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message