lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sarfaraz masood <sarfarazmasood2...@yahoo.com>
Subject Problem during indexing
Date Mon, 12 Jul 2010 13:08:07 GMT
i am trying to add 20 million documents to my index from another index that contains these
documents(cant help this architecture..its something that i will have to follow) Now the problems
i am facing are the following :

1) Too many files open error.. its at the code which is adding documents to my index.

            IndexWriter w = new IndexWriter();
IndexWriter w = new IndexWriter(index , analyzer, IndexWriter.MaxFieldLength.UNLIMITED );
            w.setMergeFactor(1000);
            w.setMaxBufferedDocs(1000);
            w.setMaxMergeDocs(600000);

            for(i =0;i<reader1.numDocs();i++)
            {
                System.out.println(i);
               addDoc(w, reader1.document(i).getField(" url ").stringValue(
),reader1.document(i).getField(" content ").stringValue().replace('.',' ').replace('-',' '));   

                    
            }
            w.optimize();
            w.close();
            reader1.close();

Due to the mergefactor parameters that i have set, around 1300 .cfs files are now opened in
index, but there is only one fdt file. These files seem to be the reason for the this error.


Are these files not closed?? do i have to do IndexWriter.commit() in the loop to close these
open files?? 



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message