lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject RE: too many open files in system
Date Tue, 09 Apr 2002 14:53:41 GMT
I've indexed 250,000 items (database rows, not files) with Lucene on a
system like this:

[otis@kyle blink]$ ulimit
unlimited
[otis@kyle blink]$ tcsh
> limit
cputime 	unlimited
filesize 	unlimited
datasize 	unlimited
stacksize 	8192 kbytes
coredumpsize 	1000000 kbytes
memoryuse 	unlimited
descriptors 	1024 
memorylocked 	unlimited
maxproc 	8192 
openfiles 	1024


I didn't encounter the 'too many open files' error.
You could change your code to optimize index after adding every X
documents to the index.
If you add Documents to a RAMDirectory-based IndexWriter, and them use
IndexWriter's addIndexes method, you will have an optimized index after
calling addIndexes method, plus you'll speed up indexing.

Otis


--- "Nader S. Henein" <nsh@bayt.net> wrote:
> The issue is the same with Lucene when you index, if you're
> indexing 200 000 files the amount of files created by the index cause
> the
> index system to run out of file handles, is there an equation to find
> out
> how many files will be created by the indexer based on the number of
> files
> we want indexed ? , becasue that would give me an idea of how many
> file
> handles
> I would need
> 
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, April 09, 2002 6:42 PM
> To: Lucene Users List; ian@plusfour.org
> Subject: RE: too many open files in system
> 
> 
> This sounds like a question for Cocoon people, as what you are asking
> about seems to be related to Cocoon's usage of Lucene, not the core
> Lucene API.
> 
> Otis
> 
> --- Ian Forsyth <ian@plusfour.org> wrote:
> > I'm calling in response to the LuceneCocconIndexer, is this class
> an
> > XML
> > file indexer? (excuse my ignorance i am just stepping into this
> whole
> > thing..)
> >
> > I do a lot of development with PHP, on different platforms
> > (WIN,*NIXES) and
> > I want to get into indexing data...
> >
> > I am wondering if there are classes existing that index XML files,
> > and for
> > search results, spit out an xml stream..
> >
> > I am figuring I would send results to lucene via the cmd line, but
> i
> > am not
> > sure how i would capture the xml results.. Do I have to have Lucene
> > make a
> > physical XML file called for instance searchresults.xml then i set
> my
> > XSLT
> > parser to that file.. ?
> >
> > I am very curious about this whole thing and any help would be a
> > super help.
> >
> > Regards,
> > Ian
> >
> > -----Original Message-----
> > From: Nader S. Henein [mailto:nsh@bayt.net]
> > Sent: Tuesday, April 09, 2002 8:56 AM
> > To: Lucene Users List; amayen@dkf.de
> > Subject: RE: too many open files in system
> >
> >
> > that depends on how many files you're indexing .. I still have to
> > figure out
> > too what logic does the LuceneCocoonIndexer adhere when it is
> > creating the
> > index files
> >
> >
> > -----Original Message-----
> > From: root [mailto:amayen@dkf.de]
> > Sent: Tuesday, April 09, 2002 4:50 PM
> > To: Lucene Users List
> > Subject: Re: too many open files in system
> >
> >
> > On Tuesday, 9. April 2002 14:08, you wrote:
> > > root wrote:
> > > > Doesn't Lucene releases the filehandles??
> > > >
> > > > because I get "too many open files in system" after running
> > lucene a
> > > > while!
> > >
> > > Are you closing the readers and writers after you've finished
> using
> > them?
> > >
> > > cheers,
> > >
> > > Chris
> >
> >
> > Yes I close the readers and writers!
> >
> >
> > @Nader S. Henein
> >
> > If I increase the filehandles, to what count should I increase
> them?
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-user-help@jakarta.apache.org>
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-user-help@jakarta.apache.org>
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> > <mailto:lucene-user-help@jakarta.apache.org>
> >
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Tax Center - online filing with TurboTax
> http://taxes.yahoo.com/
> 
> --
> To unsubscribe, e-mail:
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
> 
> 
> 
> --
> To unsubscribe, e-mail:  
> <mailto:lucene-user-unsubscribe@jakarta.apache.org>
> For additional commands, e-mail:
> <mailto:lucene-user-help@jakarta.apache.org>
> 


__________________________________________________
Do You Yahoo!?
Yahoo! Tax Center - online filing with TurboTax
http://taxes.yahoo.com/

--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message