lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Blase <mblas...@gmail.com>
Subject Re: old fashioned....."Too many open files"!
Date Fri, 18 May 2012 18:30:02 GMT
Ian was right! I didn't notice that before each insert the code was
performing a search!

but I'm not sure how to solve the problem! This is how I changed the code,
after each search I'm closing the IndexSearcher....but still....I get too
many open files!


 private IndexSearcher getSearcher() throws CorruptIndexException,
IOException {
        int NumberOfIndexes = Writers.size();

        ArrayList<IndexReader> readers = new ArrayList<IndexReader>();
        IndexReader[] readerList = new IndexReader[NumberOfIndexes];

        Set set = Writers.entrySet();
        Iterator i = set.iterator();
        while(i.hasNext()){
            Map.Entry index = (Map.Entry)i.next();
            IndexWriter iw = (IndexWriter)index.getValue();
            readers.add(IndexReader.open(iw, true));
        }

        MultiReader mr = new MultiReader(readers.toArray(readerList));
        return new IndexSearcher(mr);
    }

public TopDocs Search(String q,Analyzer analyzer,int NumberOfResults)
throws Exception {
        ExtendedQueryParser parser = new
ExtendedQueryParser(LuceneVersion.CurrentVersion,"ID",analyzer);
Query query = parser.parse(q);

        IndexSearcher is = getSearcher();
        TopDocs res = is.search(query, NumberOfResults);
        is.close();

return res;
    }




On Fri, May 18, 2012 at 11:04 AM, Edward W. Rouse <erouse@comsquared.com>wrote:

> I don't know. I do it as a matter of course. But if it fixes the problem,
> then at least you know why you are getting the error and can work on a
> scheme (using counters maybe), to do regular commits after every 10/20/100
> documents.
>
> But you can't fix it until you know why it happens and this would confirm
> or
> eliminate one possible cause.
>
> > -----Original Message-----
> > From: Michel Blase [mailto:mblase23@gmail.com]
> > Sent: Friday, May 18, 2012 1:49 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: old fashioned....."Too many open files"!
> >
> > but commit after each insert should be really expensive and
> > unnecessary! no?
> >
> > On Fri, May 18, 2012 at 10:31 AM, Edward W. Rouse
> > <erouse@comsquared.com>wrote:
> >
> > > Have you tried adding im.commit() after adding a document? Could be
> > all of
> > > the uncommitted documents are leaving files open.
> > >
> > > > -----Original Message-----
> > > > From: Michel Blase [mailto:mblase23@gmail.com]
> > > > Sent: Friday, May 18, 2012 1:24 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: old fashioned....."Too many open files"!
> > > >
> > > > also.....my problem is indexing!
> > > >
> > > > Preparation:
> > > >
> > > > private void SetUpWriters() throws Exception {
> > > >         Set set = IndexesPaths.entrySet();
> > > >         Iterator i = set.iterator();
> > > >
> > > >         while(i.hasNext()){
> > > >             Map.Entry index = (Map.Entry)i.next();
> > > >             int id = (Integer)index.getKey();
> > > >             String path = (String)index.getValue();
> > > >
> > > >             File app = new File(path);
> > > >             Directory dir = FSDirectory.open(app);
> > > >             IndexWriterConfig config = new
> > > > IndexWriterConfig(LuceneVersion.CurrentVersion,new
> > > > StandardAnalyzer(LuceneVersion.CurrentVersion));
> > > >
> > > >             //config.setMaxBufferedDocs(50);
> > > >             config.setRAMBufferSizeMB(400);
> > > >             TieredMergePolicy mp =
> > > > (TieredMergePolicy)config.getMergePolicy();
> > > >             mp.setUseCompoundFile(true);
> > > >             config.setMergePolicy(mp);
> > > >
> > > >             /*
> > > >             LogMergePolicy lmp =
> > > > (LogMergePolicy)config.getMergePolicy();
> > > >             lmp.setUseCompoundFile(true);
> > > >             lmp.setMaxMergeDocs(10000);
> > > >             config.setMergePolicy(lmp);
> > > >             */
> > > >
> > > >             Writers.put(id, new IndexWriter(dir,config));
> > > >         }
> > > >     }
> > > >
> > > >
> > > > adding document:
> > > >
> > > > public void AddDocument(Document doc,Analyzer analyzer) throws
> > > > CorruptIndexException, IOException {
> > > >         IndexWriter im = Writers.get(this.CurrentOpenIndex_ID);
> > > >         im.addDocument(doc, analyzer);
> > > >     }
> > > >
> > > >
> > > > there's not much more I'm doing!
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message