lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Index.close() infinite TIME_WAITING (repost)
Date Fri, 09 Oct 2009 10:15:29 GMT
Were there any exceptions inside Lucene, before the hang?

The fact that you're hitting AlreadyClosedException is a spooky sign
-- that means IW thinks you had in fact closed the writer, but then
used it again.

For increasing indexing throughput, I'd start here:

    http://wiki.apache.org/lucene-java/ImproveIndexingSpeed

Can you whittle down your code to a smaller test that still shows the hang?

Mike

On Fri, Oct 9, 2009 at 6:05 AM, Jamie Band <jamie@stimulussoft.com> wrote:
> Hi Mike
>
> There are other threads involved but none are simultaneously modifying the
> index.
>
> There is one thread that retrieves the total count every 2 seconds on the
> index for GUI display:
>
> public long getTotalMessageCount(Volume volume) throws
> MessageSearchException {
>             if (volume == null)
>                   throw new MessageSearchException("assertion failure: null
> volume",logger);
>             int count = 0;
>             File indexDir = new File(volume.getIndexPath());
>             if (!indexDir.exists())
>                     return 0;
>             IndexReader indexReader = null;
>             try {
>                 indexReader = IndexReader.open(indexDir);
>                 count += indexReader.numDocs();
>             } catch (IOException e ) {
>                 logger.error("failed to close index to calculate total
> count",e);
>             } finally {
>                 try { indexReader.close(); } catch (Exception e) {
>                     logger.error("failed to close index to calculate total
> count",e);
>                 }
>             }
>             return count;
> }
>
> There are also other threads for performing searches across the various
> indexes.
>
> I will rerun the reindex and send you the log.
>
> With the availability of Lucene 2.9.0 I changed our indexing routines, so
> perhaps its an issue with my code. One thing I've already been curious about
> is how it is possible to receive the AlreadyClosedException when there is no
> other code writing to the index due to the indexLock. Very occasionally  an
>  AlreadyClosedException is thrown while adding a document to the index. To
> ensure that nothing is list, a separate retry queue is maintained.
>
> Any you spot anything suspect in the below? I would also be interested to
> get your thoughts on how to improve indexing speed. Using a thread pool to
> addDocuments to the index does not seem to help much.
>
>
> public class IndexProcessor extends Thread {
>                     public IndexProcessor() {
>               setName("index processor");
>           }
>               public void run() {
>               boolean exit = false;
>               //ExecutorService documentPool;
>               // we abandoned pool as it does not seem to offer any major
> performance benefit
>               IndexInfo indexInfo = null;
> LinkedList<IndexInfo> pushbacks = new LinkedList<IndexInfo>();
>                             while (!exit) {
>                                     try {
>                       int maxIndexDocs =
> Config.getConfig().getIndex().getMaxSimultaneousDocs();
>                       //documentPool =
> Executors.newFixedThreadPool(Config.getConfig().getArchiver().getArchiveThreads());
>                       indexInfo = null;                        
indexInfo =
> (IndexInfo) queue.take();
>                       if (indexInfo==EXIT_REQ) {
>                           logger.debug("index exit req received. exiting");
>                           exit = true;
>                           continue;
>                       }
>                                         indexLock.lock();
>                       try {
>                            openIndex();
>                       } catch (Exception e) {
>                            logger.error("failed to open
> index:"+e.getMessage(),e);
>                            return;
>                       }
>                       if (indexInfo==null) {
>                           logger.debug("index info is null");
>                       }
>                       int i = 0;
>                       while(indexInfo!=null && i<maxIndexDocs)
{
>                           try {
>                               writer.addDocument(indexInfo.getDocument());
>                               indexInfo.cleanup();
>                           } catch (IOException io) {
>                               logger.error("failed to add document to
> index:"+io.getMessage(),io);
>                               indexInfo.cleanup();
>                           } catch (AlreadyClosedException e) {
>                               pushbacks.add(indexInfo);
>                           }
>                             //documentPool.execute(new
> IndexDocument(indexInfo,pushbacks));
>                                                      i++;
>                            if (i<maxIndexDocs) {
>                                indexInfo = (IndexInfo) queue.poll();
>                                                          
     if
> (indexInfo==null) {
>                                       logger.debug("index info is
null");
>                                }
>                                                          
     if
> (indexInfo==EXIT_REQ) {
>                                        logger.debug("index exit req
> received. exiting (2)");
>                                       exit = true;
>                                       break;
>                                 }
>                            }
>                                                   }
>                       if (pushbacks.size()>0) {
>                             closeIndex();
>                             try {
>                                    openIndex();
>                             } catch (Exception e) {
>                                logger.error("failed to open
> index:"+e.getMessage(),e);
>                                return;
>                             }
>                             for (IndexInfo pushback : pushbacks) {
>                                   try {
>
> writer.addDocument(pushback.getDocument());
>                                       pushback.cleanup();
>                                   } catch (IOException io) {
>                                       logger.error("failed to add
document
> to index:"+io.getMessage(),io);
>                                       pushback.cleanup();
>                                   } catch (AlreadyClosedException e)
{
>                                       pushbacks.add(pushback);
>                                   }
>                                   //documentPool.execute(new
> IndexDocument(pushback,pushbacks));
>                                   i++;
>                             }
>                       }
>                                             //documentPool.shutdown();
>                       //documentPool.awaitTermination(30,TimeUnit.MINUTES);
>                                          } catch (Throwable ie)
{
>                        logger.error("index write
> interrupted:"+ie.getMessage());
>                    } finally {
>                          closeIndex();
>                         indexLock.unlock();
>                   }
>               }                }
>
> Many thanks
>
> Jamie
>
> Michael McCandless wrote:
>>
>> Are there other threads involved, besides the one hung in close?  Can
>> you post their stack traces?
>>
>> This stack trace seems to indicate that IW believes another thread is
>> in the process of closing.
>>
>> Can you call IndexWriter.setInfoStream and post the output leading to the
>> hang?
>>
>> Mike
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message