lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vivek sar" <vivex...@gmail.com>
Subject Re: DefaultIndexAccessor
Date Sat, 16 Feb 2008 02:02:22 GMT
Mark,

  Here is the scenario when I saw this exception,

1) A search was run which uses MultiSearcher. This search took more
than 3 mins to complete (due to index size and multiple indices)
2) Just a minute after the search was started, we started writing (in
a separate thread) to one of the index which was searched on
3) Writer finished in few seconds and called writer.release
4) Since, the multi-searcher was still running the writer.release
waits for all multisearcher to complete
5) The MultiSearcher finally completes and calls
MultiIndexAccessor.release for the multisearcher
6) At this point for some reason the multisearcher throws NPE. I'm not
sure whether the NPE was on the index that was being updated by writer
or some other index
7) Because of NPE the index that the writer was writing to never gets
released and Writer gets stuck

I haven't been able to reproduce this again.

Is IndexAccessor-02.07.2008.zip
(https://issues.apache.org/jira/browse/LUCENE-1026) most up to date
code you got? You mentioned the new jar should,

"Releasing a Writer never blocks for a reopen now - so after adding a
doc it may be a second or two before its visible to new Searchers"

 Do you think this would help the case I ran into where the writer was
stuck because of the searcher release?

 I think you may still want to check for null in the
MultiIndexAccessor.release(), i.e.,

    public synchronized void release(Searcher multiSearcher) {
    Searchable[] searchers = ((MultiSearcher) multiSearcher).getSearchables();
    IndexAccessor accessor = null;
    for (Searchable searchable : searchers) {
      if(searchable != null){
         accessor = multiSearcherAccessors.remove(searchable);
         if(accessor != null){
               accessor.release((Searcher) searchable);
         }
      }
    }
}

Thanks,
-vivek

On Feb 15, 2008 5:02 PM, Mark Miller <markrmiller@gmail.com> wrote:
> Hey vivek, sorry to hear you are having problems.
>
> I am trying to figure out how you may be seeing this problem. The
> IndexAccessor cannot return null because you would get an
> IllegalStateException not a NullPointerException. Also, the released
> MultiSearcher cannot be null because the Exception would have been
> thrown sooner. Releasing a null Searcher throws no Exception. So a
> possibility is that you are returning a foreign MultiSearcher?
>
> Unlikely, but I don't see anything else at the moment.
>
> The MultiSearcher code is really pretty simple and actually recreates a
> MultiSearcher on every request...it did not appear to be worth it to
> coordinate closed sub Accessors with a cache for the MultiSearcher (I
> wrote the code at one point, and later got rid of it). So really the
> MultiSearcher is just a simple class that gets cached sub Searchers for
> each index and creates a one time use MultiSearcher. A simple cache is
> kept around that identifies which Accessor needs to release which sub
> Searcher. It's all rather simple, and I am struggling to see another
> possibility beyond returning a foreign MultiSearcher somehow.
>
> I will keep looking and keep you posted. In the mean time, do you have
> any other data or code snippets to share?
>
>
> vivek sar wrote:
> > Mark,
> >
> >    There seems to be some issue with DefaultMultiIndexAccessor.java. I
> > got following NPE exception,
> >
> >      2008-02-13 07:10:28,021 ERROR [http-7501-Processor6] ReportServiceImpl -
> > java.lang.NullPointerException
> >         at org.apache.lucene.indexaccessor.DefaultMultiIndexAccessor.release(DefaultMultiIndexAccessor.java:89)
> >
> > Looks like the IndexAccessor for one of the Searcher in the
> > MultiSearcher returned null. Not sure how is that possible, any ideas
> > how is that possible?
> >
> > In my case it caused a critical error as the writer thread was stuck
> > forever (we found out after couple of days) because of this,
> >
> > "PS thread 9" prio=1 tid=0x00002aac70eb95d0 nid=0x6ba in Object.wait()
> > [0x0000000047533000..0x0000000047533b80]
> >         at java.lang.Object.wait(Native Method)
> >         - waiting on <0x00002aab3e5c7700> (a
> > org.apache.lucene.indexaccessor.DefaultIndexAccessor)
> >         at java.lang.Object.wait(Unknown Source)
> >         at org.apache.lucene.indexaccessor.DefaultIndexAccessor.waitForReadersAndCloseCached(DefaultIndexAccessor.java:593)
> >         at org.apache.lucene.indexaccessor.DefaultIndexAccessor.release(DefaultIndexAccessor.java:510)
> >         - locked <0x00002aab3e5c7700> (a
> > org.apache.lucene.indexaccessor.DefaultIndexAccessor)
> >
> > The only way to recover was to re-start the application.
> >
> > I use both MultiSearcher and IndexSearcher in my application, I've
> > looked at your code but not able to pinpoint how can it go wrong? Of
> > course, you do have to check for null in the
> > MultiIndexAccessor.release, but how could you get null index accessor
> > at first place?
> >
> > I do call IndexAccessor.close during partitioning of indexes, but the
> > close should wait for all Searchers to close before doing anything.
> >
> > Do you have any updates to your code since 02/04/2008?
> >
> > Thanks,
> > -vivek
> >
> > On Feb 6, 2008 8:37 AM, Jay <yu@ai.sri.com> wrote:
> >
> >> Thanks for your clarifications, Mark!
> >>
> >>
> >> Jay
> >>
> >>
> >> Mark Miller wrote:
> >>
> >>>> 5. Although currently IndexSearcher.close() does almost nothing except
> >>>> to close the internal index reader, it might be a safer to close
> >>>> searcher itself as well in closeCachedSearcher(), just in case, the
> >>>> searcher may have other resources to release in the future version of
> >>>> Lucene.
> >>>>
> >>> Didn't catch that "as well". You are right, great idea Jay, thanks.
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message