lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Call <brian.c...@soterawireless.com>
Subject Re: AlreadyClosedException on new index
Date Tue, 06 Jan 2015 18:48:19 GMT
Hi Guys,

So I think I’ve figured it out… We use a common merge scheduler to handle merges across
all of our indices. When the SearchManager starts up, it triggers a merge operation, this
merge request pushes a task on the merge scheduling queue and we only allow one merge request
per index writer in the queue at a time. What appears to be happening is that a closed index
is being left in the merge queue and the call is failing due to the .equals() method on the
closed index when inserting into our BlockingArrayQueue. Usually this shouldn’t happen and
appears to be a timing issue combined with not protecting against this possibility. 

Thanks for your assistance guys!

-Brian  


> On Jan 6, 2015, at 10:22 AM, Brian Call <brian.call@soterawireless.com> wrote:
> 
> No exception at all… and that’s the crazy part. I create a new IndexWriter and then
immediately create a new SearcherManager using that freshly-created IndexWriter. Granted,
I don’t actually do anything with the IndexWriter before passing it to the SearcherManager,
but no exceptions are thrown either. 
> 
> -Brian
> 
> 
>> On Jan 6, 2015, at 9:55 AM, Ian Lea <ian.lea@gmail.com <mailto:ian.lea@gmail.com>>
wrote:
>> 
>> Presumably no exception is thrown from the new IndexWriter(....) call?
>> I'd double check that, and try some harmless method call on the
>> writer and make sure that works.  And run CheckIndex against the
>> index.
>> 
>> 
>> --
>> Ian.
>> 
>> 
>> On Tue, Jan 6, 2015 at 5:05 PM, Brian Call
>> <brian.call@soterawireless.com <mailto:brian.call@soterawireless.com>>
wrote:
>>> Hi Tomoko,
>>> 
>>> Thank you for your response! We’ve actually never seen this before in the three+
years of developing using Lucene 3.6.x. The only time we’ve ever seen this kind of exception
is once recently in a running production system and it caught me way off guard. We’re deploying
on Suse linux (enterprise), and jdk1.7.
>>> 
>>> Our application creates and deletes indices within the context of a single thread
so I don’t think another thread is abruptly closing the index. All index open/close operations
are always done sequentially in the context of one thread as operation requests are received.
>>> 
>>> Blessings,
>>> Brian
>>> 
>>> 
>>> 
>>>> On Jan 6, 2015, at 2:16 AM, Tomoko Uchida <tomoko.uchida.1111@gmail.com
<mailto:tomoko.uchida.1111@gmail.com>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> How often does this error occur?
>>>> You do not tell the lucene version, but I guess you use lucene 3.x
>>>> according to the stack trace...
>>>> IndexWriter would not be closed until IndexWriter.close() method is called
>>>> explicitly.
>>>> https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
<https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java>
<https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
<https://github.com/apache/lucene-solr/blob/lucene_solr_3_6_2/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java>>
>>>> 
>>>> Do you have custom codes wrapping lucene objects? There can be any codes
>>>> which call IndexWriter.close() unexpectedly?
>>>> 
>>>> If your application seems to have no problem, you need to share more
>>>> information including lucene version and system environments.
>>>> 
>>>> Regards,
>>>> Tomoko
>>>> 
>>>> 
>>>> 2015-01-06 8:32 GMT+09:00 Brian Call <brian.call@soterawireless.com <mailto:brian.call@soterawireless.com>
<mailto:brian.call@soterawireless.com <mailto:brian.call@soterawireless.com>>>:
>>>> 
>>>>> Hi Guys,
>>>>> 
>>>>> So I’m seeing a problem in production that is very bizarre and I wanted
to
>>>>> see if anyone else has encountered this issue and/or has a suggestion
or
>>>>> fix. Here goes:
>>>>> 
>>>>> We create a wrapper around an index and searcher manager to encapsulate
>>>>> both. First we create the IndexWriter and then immediately after create
the
>>>>> SearcherManager, like this:
>>>>> 
>>>>> indexWriter = new IndexWriter(indexDirectory, getWriterConfig());
>>>>> searcherManager = new SearcherManager(indexWriter, true, new
>>>>> ExecutorSearcherFactory());
>>>>> 
>>>>> 
>>>>> During construction of the SearcherManager this is thrown:
>>>>> 
>>>>> Caused by: org.apache.lucene.store.AlreadyClosedException: this
>>>>> IndexWriter is closed
>>>>>       at
>>>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:766)
>>>>>       at
>>>>> org.apache.lucene.index.IndexWriter.getDirectory(IndexWriter.java:1909)
>>>>>       at
>>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.getIndexDirectory(LuceneIndexMergeScheduler.java:162)
>>>>>       at
>>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.access$000(LuceneIndexMergeScheduler.java:31)
>>>>>       at
>>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler$MergeTask.equals(LuceneIndexMergeScheduler.java:127)
>>>>>       at
>>>>> java.util.concurrent.ArrayBlockingQueue.contains(ArrayBlockingQueue.java:497)
>>>>>       at
>>>>> com.triagewireless.h1s.session.data.index.LuceneIndexMergeScheduler.merge(LuceneIndexMergeScheduler.java:148)
>>>>>       at
>>>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2740)
>>>>>       at
>>>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2734)
>>>>>       at
>>>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:457)
>>>>>       at
>>>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
>>>>>       at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
>>>>>       at
>>>>> org.apache.lucene.search.SearcherManager.<init>(SearcherManager.java:82)
>>>>>       at
>>>>> com.triagewireless.h1s.session.data.index.LuceneIndex.initNoCache(LuceneIndex.java:312)
>>>>>       at
>>>>> com.triagewireless.h1s.session.data.index.LuceneIndex.init(LuceneIndex.java:270)
>>>>>       at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
>>>>>       at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>       at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>       at
>>>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:346)
>>>>>       at
>>>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:299)
>>>>>       at
>>>>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:132)
>>>>>       ... 57 more
>>>>> 
>>>>> 
>>>>> Why? No other threads are acting on this index, etc. The index writer
was
>>>>> just created so how could it already be closed?
>>>>> 
>>>>> I’m completely baffled on this one guys… so many thanks in advance
for
>>>>> your help! I’ll take any suggestions on a possible mitigation too if
anyone
>>>>> thinks of any.
>>>>> 
>>>>> 
>>>>> Blessings,
>>>>> Brian Call
>>>>> Manager, Systems Software Development
>>>>> Work: +1 (619) 373-4840 | Cell: +1 (619) 344-1013
>>>>> 
>>>>> <http://soterawireless.com/ <http://soterawireless.com/> <http://soterawireless.com/
<http://soterawireless.com/>>>                   Sotera Wireless, Inc.
>>>>> 10020 Huennekens Street San Diego, CA 92121, USA
>>>>> Phone: +1 (858) 427-4620 | Fax: +1 (858) 999-2487
>>>>> 
>>>>> PRIVILEGED AND CONFIDENTIAL COMMUNICATION: This electronic transmission,
>>>>> and any documents attached hereto, may contain confidential and/or legally
>>>>> privileged information. The information is intended only for use by the
>>>>> recipient named above. If you have received this electronic message in
>>>>> error, please notify the sender and delete the electronic message. Any
>>>>> disclosure, copying, distribution, or use of the contents of information
>>>>> received in error is strictly prohibited.
>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org <mailto:java-user-unsubscribe@lucene.apache.org>
>> For additional commands, e-mail: java-user-help@lucene.apache.org <mailto:java-user-help@lucene.apache.org>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message