lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Chan" <dayz...@gmail.com>
Subject Re: Running 20mil queries against an index
Date Tue, 23 May 2006 10:05:04 GMT
Thanks for that. Does anyone know how much RAM a 5gb index will need?
With mx set to 27gb, it crashes when it flushes buffer at one point.

"bash-2.03$ Exception in thread "main" java.lang.ExceptionInInitializerError
        at TaxonomyFinder.RelatedCatsFinder.<init>(RelatedCatsFinder.java:46)
        at wikipedia.WikipediaAnalyser$ExtractAbstractHandler.endElement(WikipediaAnalyser.java:295)
        at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown
Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown
Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at wikipedia.WikipediaAnalyser.parseAbstracts(WikipediaAnalyser.java:184)
        at wikipedia.WikipediaAnalyser.getRelatedCategories(WikipediaAnalyser.java:127)
        at TaxonomyFinder.TaxonomyTreeMaker.main(TaxonomyTreeMaker.java:492)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -2097152
        at java.util.Vector.elementAt(Unknown Source)
        at org.apache.lucene.store.RAMOutputStream.flushBuffer(RAMOutputStream.java:82)
        at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:84)
        at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:52)
        at org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:68)
        at org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:95)
        at word_coocurrence.WordCooccurrenceFinder.<clinit>(WordCooccurrenceFinder.java:50)
        ... 13 more"

I build the RAMDirectory from a FSDirectory, using the following:

		try {
			m_searcher = new IndexSearcher(new RAMDirectory("index"));
		} catch (IOException e) {
			e.printStackTrace();
			System.exit(1);
		}

where "index" is the path for the index.

Any help will be much appreciated.

Michael

On 5/23/06, Daniel Naber <lucenelist2005@danielnaber.de> wrote:
> On Dienstag 23 Mai 2006 08:26, Michael Chan wrote:
>
> > As I have quite a
> > bit of RAM (~20gb), is there a way I could store the index in RAM or
> > any other way that makes use of it to improve performance?
>
> RAMDirectory has just been fixed (in SVN) to work with indexes > 2 GB.
>
> Regards
>  Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message