lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahil <qam...@cs.man.ac.uk>
Subject OutOfMemory and IOException Access Denied errors
Date Fri, 19 May 2006 12:13:55 GMT
Hi

I am new to Lucene so am perhaps missing something obvious. I have included Lucene 1.9.1 in
my classpath and am trying to integrate it with MySQL.

I have a table which has near a million records in it. According to the documentation on Lucene
I have read so far, my understanding is that I need to (1) make a connection with MySQL then
(2) execute the query normally in SQL syntax. (3) Then pass the ResultSet to the method to
create indexes. (4) I can then pass a queryString to the searchIndex() custom method to locate
the queryString. 

PROBLEMS: 

(a) The first problem I had was when trying to execute the query on the million records table
in Step 1. It resulted in an OutOfMemory error due to the size of the table. How can I get
around this problem so that the query executes on the entire table at one time? 

As a workaround, I limited the number of results to 100,000 which worked fine.

(b) But I then received an IOException when the index was being written to the Document object.
The Exception stack trace is shown below: 

---
Exception in thread "main" java.io.IOException: Access is denied
	at java.io.WinNTFileSystem.createFileExclusively(Native Method)
	at java.io.File.createNewFile(File.java:850)
	at org.apache.lucene.store.FSDirectory$1.obtain(FSDirectory.java:324)
	at org.apache.lucene.store.Lock.obtain(Lock.java:92)
	at org.apache.lucene.store.Lock$With.run(Lock.java:147)
	at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:442)
	at org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:401)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:260)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:244)
	at man.ac.uk.most.LuceneIndex.indexResultSet(LuceneIndex.java:102)  --- error line in my
piece of code !
	at man.ac.uk.most.LuceneIndex.main(LuceneIndex.java:40)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:78)

---

Line 102 is present in the block of code in my program as such

----
while(rs.next()){
                lucDoc = new Document();
                lucDoc.add(Field.Keyword("conceptId",rs.getString("CONCEPTID")));
                lucDoc.add(Field.Text("term",rs.getString("TERM")));
                lucDoc.add(Field.UnIndexed("descriptionType",rs.getString("DESCRIPTIONTYPE")));

                indexWriter.addDocument(lucDoc);		--- problem line 102
            }

rs.close();
closeIndexWriter();


----

If I limit Step 1 to execute 10000 records then the program runs fine and theres no problem.
However I need to index the entire table either as a single query or an incremental query.

Can someone please help me with these problems.

Thanks
Rahil


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message