Return-Path: X-Original-To: apmail-lucene-java-user-archive@www.apache.org Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 72FF8C5BA for ; Fri, 18 May 2012 16:59:00 +0000 (UTC) Received: (qmail 76396 invoked by uid 500); 18 May 2012 16:58:58 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 76343 invoked by uid 500); 18 May 2012 16:58:58 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 76335 invoked by uid 99); 18 May 2012 16:58:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 May 2012 16:58:57 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ian.lea@gmail.com designates 209.85.213.176 as permitted sender) Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 May 2012 16:58:53 +0000 Received: by yenm14 with SMTP id m14so3939379yen.35 for ; Fri, 18 May 2012 09:58:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=/MY+TBP5xTX8hQ8gV8V4i8v1lCSHXwFvgUk6QcH/XFQ=; b=I0mrZGzvSL0GFg3VwkAwEXVj7Mopbf8evhCssjSAPIXmiBpifU54VOi9DrHG0eS0tI GkueEmnLXbmnaVdY9Ry69P8bth3lvYQyCYhRhJt3oUhriwYz/m2pUxj5n77HiScFUKv3 xdL6+xz7/hWZpHLnZlxTnSKRtBtsGsEoeMFq+KXDRenmIvlsWXzMgeTNttPN2EgEDCWs Drt/2wSqN+BIwMVHOn7peZpVod+E69Z4Bb0sE7dnyodhtYaPjXjUIEVnZ+T/EvKQAIu1 B8ucKuuiwkre8vASfzUKkC0wwYy9M6udWmaw8uQLpZU2+Ng3DF3cIprN6jGuDgkKevOb 0RdQ== Received: by 10.50.51.227 with SMTP id n3mr1311879igo.12.1337360312064; Fri, 18 May 2012 09:58:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.102.197 with HTTP; Fri, 18 May 2012 09:58:11 -0700 (PDT) In-Reply-To: References: From: Ian Lea Date: Fri, 18 May 2012 17:58:11 +0100 Message-ID: Subject: Re: old fashioned....."Too many open files"! To: java-user@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org You may need to cut it down to something simpler, but I can't see any reader.close() calls. -- Ian. On Fri, May 18, 2012 at 5:47 PM, Michel Blase wrote: > This is the code in charge of managing the Lucene index. Thanks for your > help! > > > > package luz.aurora.lucene; > > import java.io.File; > import java.io.IOException; > import java.util.*; > import luz.aurora.search.ExtendedQueryParser; > import org.apache.lucene.analysis.Analyzer; > import org.apache.lucene.analysis.standard.StandardAnalyzer; > import org.apache.lucene.document.Document; > import org.apache.lucene.index.*; > import org.apache.lucene.queryParser.ParseException; > import org.apache.lucene.search.IndexSearcher; > import org.apache.lucene.search.Query; > import org.apache.lucene.search.TopDocs; > import org.apache.lucene.search.highlight.Highlighter; > import org.apache.lucene.search.highlight.QueryScorer; > import org.apache.lucene.search.highlight.SimpleHTMLFormatter; > import org.apache.lucene.search.highlight.SimpleSpanFragmenter; > import org.apache.lucene.store.Directory; > import org.apache.lucene.store.FSDirectory; > > > public class LuceneManager { > > =A0 =A0private HashMap IndexesPaths; > =A0 =A0private HashMap Writers; > > =A0 =A0private int CurrentOpenIndex_ID; > =A0 =A0private String CurrentOpenIndex_TablePrefix; > > =A0 =A0public =A0LuceneManager(int CurrentOpenIndex_ID,String > CurrentOpenIndex_TablePrefix, HashMap IndexesPaths) throw= s > Exception { > =A0 =A0 =A0 =A0this.CurrentOpenIndex_ID =3D CurrentOpenIndex_ID; > =A0 =A0 =A0 =A0this.IndexesPaths =3D IndexesPaths; > =A0 =A0 =A0 =A0this.Writers =3D new HashMap(); > =A0 =A0 =A0 =A0this.CurrentOpenIndex_TablePrefix =3D CurrentOpenIndex_Tab= lePrefix; > > =A0 =A0 =A0 =A0SetUpWriters(); > =A0 =A0} > > =A0 =A0private void SetUpWriters() throws Exception { > =A0 =A0 =A0 =A0Set set =3D IndexesPaths.entrySet(); > =A0 =A0 =A0 =A0Iterator i =3D set.iterator(); > > =A0 =A0 =A0 =A0while(i.hasNext()){ > =A0 =A0 =A0 =A0 =A0 =A0Map.Entry index =3D (Map.Entry)i.next(); > =A0 =A0 =A0 =A0 =A0 =A0int id =3D (Integer)index.getKey(); > =A0 =A0 =A0 =A0 =A0 =A0String path =3D (String)index.getValue(); > > =A0 =A0 =A0 =A0 =A0 =A0File app =3D new File(path); > =A0 =A0 =A0 =A0 =A0 =A0Directory dir =3D FSDirectory.open(app); > =A0 =A0 =A0 =A0 =A0 =A0IndexWriterConfig config =3D new > IndexWriterConfig(LuceneVersion.CurrentVersion,new > StandardAnalyzer(LuceneVersion.CurrentVersion)); > > =A0 =A0 =A0 =A0 =A0 =A0//config.setMaxBufferedDocs(50); > =A0 =A0 =A0 =A0 =A0 =A0config.setRAMBufferSizeMB(400); > =A0 =A0 =A0 =A0 =A0 =A0TieredMergePolicy mp =3D > (TieredMergePolicy)config.getMergePolicy(); > =A0 =A0 =A0 =A0 =A0 =A0mp.setUseCompoundFile(true); > =A0 =A0 =A0 =A0 =A0 =A0config.setMergePolicy(mp); > > =A0 =A0 =A0 =A0 =A0 =A0/* > =A0 =A0 =A0 =A0 =A0 =A0LogMergePolicy lmp =3D (LogMergePolicy)config.getM= ergePolicy(); > =A0 =A0 =A0 =A0 =A0 =A0lmp.setUseCompoundFile(true); > =A0 =A0 =A0 =A0 =A0 =A0lmp.setMaxMergeDocs(10000); > =A0 =A0 =A0 =A0 =A0 =A0config.setMergePolicy(lmp); > =A0 =A0 =A0 =A0 =A0 =A0*/ > > =A0 =A0 =A0 =A0 =A0 =A0Writers.put(id, new IndexWriter(dir,config)); > =A0 =A0 =A0 =A0} > =A0 =A0} > > =A0 =A0public void AddDocument(int IndexId,Document doc,Analyzer analyzer= ) > throws CorruptIndexException, IOException { > =A0 =A0 =A0 =A0IndexWriter im =3D Writers.get(IndexId); > =A0 =A0 =A0 =A0im.addDocument(doc, analyzer); > =A0 =A0} > > =A0 =A0public void AddDocument(Document doc,Analyzer analyzer) throws > CorruptIndexException, IOException { > =A0 =A0 =A0 =A0IndexWriter im =3D Writers.get(this.CurrentOpenIndex_ID); > =A0 =A0 =A0 =A0im.addDocument(doc, analyzer); > =A0 =A0} > > =A0 =A0public void DeleteDoc(int IndexId,int SegmentIdFromDb) throws > CorruptIndexException, IOException { > =A0 =A0 =A0 =A0IndexWriter im =3D Writers.get(IndexId); > =A0 =A0 =A0 =A0Term term =3D new Term("SegmentID",Integer.toString(Segmen= tIdFromDb)); > =A0 =A0 =A0 =A0im.deleteDocuments(term); > =A0 =A0} > > =A0 =A0public void DeleteDocuments(String query) throws ParseException, > CorruptIndexException, IOException { > > =A0 =A0 =A0 =A0ExtendedQueryParser parser =3D new > ExtendedQueryParser(LuceneVersion.CurrentVersion,"ID",new > StandardAnalyzer(LuceneVersion.CurrentVersion)); > Query q =3D parser.parse(query); > > =A0 =A0 =A0 =A0Set set =3D Writers.entrySet(); > =A0 =A0 =A0 =A0Iterator i =3D set.iterator(); > > =A0 =A0 =A0 =A0while(i.hasNext()){ > =A0 =A0 =A0 =A0 =A0 =A0Map.Entry app =3D (Map.Entry)i.next(); > =A0 =A0 =A0 =A0 =A0 =A0IndexWriter im =3D (IndexWriter)app.getValue(); > =A0 =A0 =A0 =A0 =A0 =A0im.deleteDocuments(q); > =A0 =A0 =A0 =A0} > =A0 =A0} > > =A0 =A0private IndexSearcher getSearcher() throws CorruptIndexException, > IOException { > =A0 =A0 =A0 =A0int NumberOfIndexes =3D Writers.size(); > > =A0 =A0 =A0 =A0ArrayList readers =3D new ArrayList(); > =A0 =A0 =A0 =A0IndexReader[] readerList =3D new IndexReader[NumberOfIndex= es]; > > =A0 =A0 =A0 =A0Set set =3D Writers.entrySet(); > =A0 =A0 =A0 =A0Iterator i =3D set.iterator(); > =A0 =A0 =A0 =A0while(i.hasNext()){ > =A0 =A0 =A0 =A0 =A0 =A0Map.Entry index =3D (Map.Entry)i.next(); > =A0 =A0 =A0 =A0 =A0 =A0IndexWriter iw =3D (IndexWriter)index.getValue(); > =A0 =A0 =A0 =A0 =A0 =A0readers.add(IndexReader.open(iw, true)); > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0MultiReader mr =3D new MultiReader(readers.toArray(readerL= ist)); > =A0 =A0 =A0 =A0return new IndexSearcher(mr); > =A0 =A0} > > =A0 =A0public void close() throws CorruptIndexException, IOException { > =A0 =A0 =A0 =A0Set set =3D Writers.entrySet(); > =A0 =A0 =A0 =A0Iterator i =3D set.iterator(); > =A0 =A0 =A0 =A0while(i.hasNext()){ > =A0 =A0 =A0 =A0 =A0 =A0Map.Entry index =3D (Map.Entry)i.next(); > =A0 =A0 =A0 =A0 =A0 =A0IndexWriter iw =3D (IndexWriter)index.getValue(); > =A0 =A0 =A0 =A0 =A0 =A0iw.close(); > =A0 =A0 =A0 =A0} > =A0 =A0} > > =A0 =A0public void commit() throws CorruptIndexException, IOException, > Exception { > =A0 =A0 =A0 =A0Set set =3D Writers.entrySet(); > =A0 =A0 =A0 =A0Iterator i =3D set.iterator(); > =A0 =A0 =A0 =A0while(i.hasNext()){ > =A0 =A0 =A0 =A0 =A0 =A0Map.Entry index =3D (Map.Entry)i.next(); > =A0 =A0 =A0 =A0 =A0 =A0IndexWriter iw =3D (IndexWriter)index.getValue(); > =A0 =A0 =A0 =A0 =A0 =A0iw.commit(); > =A0 =A0 =A0 =A0} > > =A0 =A0} > > =A0 =A0public int getCurrentOpenIndexId() { > =A0 =A0 =A0 =A0return this.CurrentOpenIndex_ID; > =A0 =A0} > > =A0 =A0public String getCurrentOpenIndexTablePrefix() { > =A0 =A0 =A0 =A0return this.CurrentOpenIndex_TablePrefix; > =A0 =A0} > > > =A0 =A0//SEARCH START > > =A0 =A0public TopDocs Search(String q,Analyzer analyzer,int NumberOfResul= ts) > throws Exception { > =A0 =A0 =A0 =A0ExtendedQueryParser parser =3D new > ExtendedQueryParser(LuceneVersion.CurrentVersion,"ID",analyzer); > Query query =3D parser.parse(q); > =A0 =A0 =A0 =A0//Filter f =3D NumericRangeFilter.newIntRange("SegmentID",= 393, 393, > true, true); > //FilteredQuery fq =3D new FilteredQuery(query,f); > //TopDocs docs =3D searcher.search(fq, NumberOfResults); > //System.out.println(searcher.getIndexReader().) > return getSearcher().search(query, NumberOfResults); > =A0 =A0} > > =A0 =A0public TopDocs SearchAndHighlight(String q,Analyzer analyzer,int > NumberOfResults) throws Exception { > =A0 =A0 =A0 =A0ExtendedQueryParser parser =3D new > ExtendedQueryParser(LuceneVersion.CurrentVersion,"ID",analyzer); > Query query =3D parser.parse(q); > =A0 =A0 =A0 =A0return getSearcher().search(query, NumberOfResults); > =A0 =A0} > > =A0 =A0public Highlighter getHighlighter(String query,Analyzer analyzer,S= tring > OpeningTag,String ClosingTag) throws ParseException { > =A0 =A0 =A0 =A0ExtendedQueryParser qp =3D new > ExtendedQueryParser(LuceneVersion.CurrentVersion,"ID",analyzer); > =A0 =A0 =A0 =A0Query q =3D qp.parse(query); > =A0 =A0 =A0 =A0QueryScorer qs =3D new QueryScorer(q); > =A0 =A0 =A0 =A0SimpleHTMLFormatter formatter =3D =A0new > SimpleHTMLFormatter(OpeningTag,ClosingTag); > =A0 =A0 =A0 =A0Highlighter hl =3D new Highlighter(formatter,qs); > =A0 =A0 =A0 =A0hl.setTextFragmenter(new SimpleSpanFragmenter(qs)); > =A0 =A0 =A0 =A0return hl; > =A0 =A0} > > =A0 =A0public Document getDoc(int i) throws CorruptIndexException, IOExce= ption > { > =A0 =A0 =A0 =A0return getSearcher().doc(i); > =A0 =A0} > > > > } > > > > > > On Thu, May 17, 2012 at 10:37 PM, findbestopensource < > findbestopensource@gmail.com> wrote: > >> Post complete code. You are not closing the objects (IndexWriter / Index >> Searcher) properly. >> >> Regards >> Aditya >> www.findbestopensource.com >> >> >> On Fri, May 18, 2012 at 6:51 AM, Michel Blase wrote= : >> >> > Hi all, >> > >> > I have few problems Indexing. I keep hitting "Too many open files". It >> > seems like Lucene is not releasing files handlers after deleting >> segments. >> > >> > This is a piece from the lsof output showing the problem: >> > >> > >> > java =A0 =A023024 root *347r =A0 =A0 =A0REG =A0 =A0 =A0251,0 =A0 =A0 2= 660 149376 >> > /home/INDEXES_ROOT/SMPL_1/_bvq.cfs (deleted) >> > java =A0 =A023024 root *348r =A0 =A0 =A0REG =A0 =A0 =A0251,0 =A0 =A0 2= 477 149382 >> > /home/INDEXES_ROOT/SMPL_1/_bvr.cfs (deleted) >> > java =A0 =A023024 root *349r =A0 =A0 =A0REG =A0 =A0 =A0251,0 =A0 =A0 2= 747 149392 >> > /home/INDEXES_ROOT/SMPL_1/_bvu.cfs (deleted) >> > java =A0 =A023024 root *350r =A0 =A0 =A0REG =A0 =A0 =A0251,0 =A0 =A0 2= 339 149384 >> > /home/INDEXES_ROOT/SMPL_1/_bvs.cfs (deleted) >> > >> > and this is the code I'm using (I'm using Lucene3.6) >> > >> > -- IndexWriter creation: >> > >> > File app =3D new File(path); >> > Directory dir =3D FSDirectory.open(app); >> > IndexWriterConfig config =3D new >> > IndexWriterConfig(LuceneVersion.CurrentVersion,new >> > StandardAnalyzer(LuceneVersion.CurrentVersion)); >> > >> > //these are random tries attempting to solve the problem: >> > config.setRAMBufferSizeMB(400); >> > TieredMergePolicy mp =3D (TieredMergePolicy)config.getMergePolicy(); >> > mp.setUseCompoundFile(true); >> > config.setMergePolicy(mp); >> > IndexWriter im =3D new IndexWriter(dir,config); >> > >> > -- Then just a loop over my doc list calling for indexing: >> > im.addDocument(doc, analyzer); >> > >> > >> > Any idea? >> > Thanks, >> > Luca >> > >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org