lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Indexing and Searching Web Application
Date Wed, 21 Jan 2009 22:32:48 GMT
NOTE: you're iterating over 'searchers' and
adding to indexSearchers. Is that a typo?

Assuming that it's not and your 'searchers'
is the copy you talk about (so you can freely
add?) you never delete from the underlying
indexSearchers. But you do close elements
because you're closing a reference to the
searcher that  points to the same underlying
object.

Assuming that's not the problem,
here's what I'd suggest. Log the count of
searchers just above your loop...

print searchers.size(); (or whatever).
for (IndexSearcher indexSearcher: searchers) {

Ian claims that the first time you'll see some number X
The second time you'll see X + Y.

You have to be getting this list of servers from someplace.
Wherever it is, it's (probably) persisted across calls,
because if it isn't you wouldn't have any open readers
to close.

Are you sure your local variable isn't just a reference
to the underlying (permanent) list?

See inline comments

for (IndexSearcher indexSearcher: searchers) {

 IndexReader reader = indexSearcher.getIndexReader();

 IndexReader newReader = reader.reopen();

 if (newReader != reader) {

  reader.close();

 }
[EOE} you have not removed the instance of the searcher from searchers (the
var in your for loop)
           but you have closed it. So next time your code tries to use it,
you've already closed it.

 reader = newReader;

 IndexSearcher indexSearch = new IndexSearcher(reader);

[EOE] This adds the newly opened searcher to the end of your array. The
original (closed) one is still there.

 indexSearchers.add(indexSearch);

}

[EOE] So if you use searchers anywhere from here on, it's got closed
readers in it if you closed any of them.

Best
Erick

On Wed, Jan 21, 2009 at 4:19 PM, Amin Mohammed-Coleman <aminmc@gmail.com>wrote:

> Hi
>
> I am trying to get an understanding and what the best practice is.
>
> I am not saying that I am right, it may well be that my code is wrong, that
> is why I am posting this.   The original loop that I am iterating over is a
> spring injected dependency.  I don't reuse that in the multisearcher.  I
> create a new list (local variable) when I invoke the search method.  So I'm
> not sure how I can be adding to an existing list.
>
> I presume it's a bad idea not to close the indexreader in this case.
>
> Cheers
>
>
>
>
>
> On 21 Jan 2009, at 20:43, Ian Lea wrote:
>
>  Oh well, it's your code so I guess you know what it does.
>>
>> But I still think you're wrong.
>>
>> If your list contains 3 searchers at the top of the loop and all 3
>> need to be reopened then the list will contain 6 searchers at the end
>> of the loop, and the first 3 will be for readers that you've just
>> closed.  Hence the already closed exception when you try to use them.
>>
>>
>> --
>> Ian.
>>
>>
>> On Wed, Jan 21, 2009 at 8:24 PM, Amin Mohammed-Coleman <aminmc@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> That is what I am doing with the line:
>>>
>>> indexSearchers.add(indexSearch);
>>>
>>> indexSearchers is an ArrayList that is constructed before the for loop:
>>>
>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>
>>>
>>> I then pass the indexSearchers to :
>>>
>>> multiSearcher = new MultiSearcher(indexSearchers.toArray(new
>>> IndexSearcher[]
>>> {}));
>>>
>>>
>>> Cheers
>>>
>>> On 21 Jan 2009, at 20:19, Ian Lea wrote:
>>>
>>>  I haven't been following this thread, but shouldn't you be replacing
>>>> the old searcher in your list of searchers rather than just adding the
>>>> new one on the end?  Could be wrong - I find the names in your code
>>>> snippet rather confusing.
>>>>
>>>>
>>>> --
>>>> Ian.
>>>>
>>>> On Wed, Jan 21, 2009 at 6:59 PM, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi
>>>>> I did the following according to java docs:
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>
>>>>> IndexReader newReader = reader.reopen();
>>>>>
>>>>> if (newReader != reader) {
>>>>>
>>>>> reader.close();
>>>>>
>>>>> }
>>>>>
>>>>> reader = newReader;
>>>>>
>>>>> IndexSearcher indexSearch = new IndexSearcher(reader);
>>>>>
>>>>> indexSearchers.add(indexSearch);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> First search works ok, susequent search result in:
>>>>>
>>>>>
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>>>> closed
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jan 21, 2009 at 1:47 PM, Amin Mohammed-Coleman
>>>>> <aminmc@gmail.com>wrote:
>>>>>
>>>>>  Hi
>>>>>> Will give that a go.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On 21 Jan 2009, at 12:26, "Ganesh" <emailgane@yahoo.co.in>
wrote:
>>>>>>
>>>>>> I am closing the old reader and it is working fine for me. Refer
to
>>>>>>
>>>>>>>
>>>>>>> IndexReader.Reopen javadoc.
>>>>>>>
>>>>>>> ///Below is the code snipper from IndexReader.reopen javadoc
>>>>>>>
>>>>>>> IndexReader reader = ...
>>>>>>> ...
>>>>>>> IndexReader new = r.reopen();
>>>>>>> if (new != reader) {
>>>>>>> ...     // reader was reopened
>>>>>>> reader.close();  //Old reader is closed.
>>>>>>> }
>>>>>>> reader = new;
>>>>>>>
>>>>>>> Regards
>>>>>>> Ganesh
>>>>>>>
>>>>>>> ----- Original Message ----- From: "Amin Mohammed-Coleman" <
>>>>>>> aminmc@gmail.com>
>>>>>>> To: <java-user@lucene.apache.org>
>>>>>>> Cc: <java-user@lucene.apache.org>
>>>>>>> Sent: Wednesday, January 21, 2009 1:07 AM
>>>>>>>
>>>>>>> Subject: Re: Indexing and Searching Web Application
>>>>>>>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> Yes I am using the reopen method on indexreader. I am not
closing
>>>>>>>> the
>>>>>>>> old indexer as per Ganesh's instruction. It seems to be working
>>>>>>>> correctly
>>>>>>>> so I presume it's ok not to close.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> Amin
>>>>>>>>
>>>>>>>> On 20 Jan 2009, at 19:27, "Angel, Eric" <eangel@business.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> There's a reopen() method in the IndexReader class.  You
can use
>>>>>>>> that.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Amin Mohammed-Coleman [mailto:aminmc@gmail.com]
>>>>>>>>> Sent: Tuesday, January 20, 2009 5:02 AM
>>>>>>>>> To: java-user@lucene.apache.org
>>>>>>>>> Subject: Re: Indexing and Searching Web Application
>>>>>>>>>
>>>>>>>>> Am I supposed to close the oldIndexReader?  I just tried
this and I
>>>>>>>>> get
>>>>>>>>> an
>>>>>>>>> exception stating that the IndexReader is closed.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> On Tue, Jan 20, 2009 at 9:33 AM, Ganesh <emailgane@yahoo.co.in>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Reopen the reader, only if it is modified.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> IndexReader oldIndexReader = indexSearcher.getIndexReader();
>>>>>>>>>> if (!oldIndexReader.isCurrent()) {
>>>>>>>>>> IndexReader newIndexReader = oldIndexReader.reOpen();
>>>>>>>>>> oldIndexReader.close();
>>>>>>>>>> indexSearcher.close();
>>>>>>>>>> IndexSearcher indexSearch = new IndexSearcher(newIndexReader);
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Ganesh
>>>>>>>>>>
>>>>>>>>>> ----- Original Message ----- From: "Amin Mohammed-Coleman"
<
>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>> To: <java-user@lucene.apache.org>
>>>>>>>>>> Sent: Tuesday, January 20, 2009 1:38 PM
>>>>>>>>>> Subject: Re: Indexing and Searching Web Application
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> After your email I had a look around and came
up with the below
>>>>>>>>>>> solution (I'm not sure if this is the right approach
or there is
>>>>>>>>>>> a
>>>>>>>>>>> performance implication to doing this)
>>>>>>>>>>>
>>>>>>>>>>> public Summary[] search(SearchRequest searchRequest)
{
>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>> MultiSearcher multiSearcher = null;
>>>>>>>>>>> List<IndexSearcher> newIndexSearchers =
new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>> try {
>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers)
{
>>>>>>>>>>> IndexReader indexReader =
>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>> IndexSearcher indexSearch = new IndexSearcher(indexReader);
>>>>>>>>>>> newIndexSearchers.add(indexSearch);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> multiSearcher = new MultiSearcher(newIndexSearchers.toArray(new
>>>>>>>>>>> IndexSearcher[] {}));
>>>>>>>>>>> QueryParser queryParser = new
>>>>>>>>>>> MultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>
>>>>>>>>>>>  analyzer);
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Query query = queryParser.parse(searchRequest.getSearchTerm());
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> //TODO: Sort and Filters
>>>>>>>>>>>
>>>>>>>>>>> TopDocs topDocs = multiSearcher.search(query,
100);
>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString()
+ "
>>>>>>>>>>> ]
>>>>>>>>>>> = " +topDocs.totalHits);
>>>>>>>>>>>
>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
score);
>>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total time taken for seach: " +
>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The searchers are configured in spring using
which looks like
>>>>>>>>>>> this:
>>>>>>>>>>>
>>>>>>>>>>> <bean id="rtfIndexSearcher"
>>>>>>>>>>> class="org.apache.lucene.search.IndexSearcher"
scope="prototype"
>>>>>>>>>>>
>>>>>>>>>>>  lazy-
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  init="true" >
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <constructor-arg type="org.apache.lucene.store.Directory"
>>>>>>>>>>> ref="rtfDirectory"  />
>>>>>>>>>>> </bean>
>>>>>>>>>>>
>>>>>>>>>>> I set the dependencies on the DocumentSearcher
class.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 19 Jan 2009, at 21:45, Amin Mohammed-Coleman
wrote:
>>>>>>>>>>>
>>>>>>>>>>> I make a call to my search class which looks
like this:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> public Summary[] search(SearchRequest searchRequest)
{
>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>> MultiSearcher multiSearcher = null;
>>>>>>>>>>>> try {
>>>>>>>>>>>> multiSearcher = new MultiSearcher(searchers.toArray(new
>>>>>>>>>>>> IndexSearcher[] {}));
>>>>>>>>>>>> QueryParser queryParser = new
>>>>>>>>>>>> MultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>> analyzer);
>>>>>>>>>>>> Query query = queryParser.parse(searchRequest.getSearchTerm());
>>>>>>>>>>>>
>>>>>>>>>>>> //TODO: Sort and Filters
>>>>>>>>>>>>
>>>>>>>>>>>> TopDocs topDocs = multiSearcher.search(query,
100);
>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>> LOGGER.debug("total number of hits for ["
+ query.toString() +
>>>>>>>>>>>> " ] = " +topDocs.totalHits);
>>>>>>>>>>>>
>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
score);
>>>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>> }
>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("total time taken for seach:
" +
>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>> return summaryList.toArray(new Summary[]
{});
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> Do I need to do this explicitly?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Amin
>>>>>>>>>>>>
>>>>>>>>>>>> On 19 Jan 2009, at 20:48, Greg Shackles wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> After you make the commit to the index, are
you reloading the
>>>>>>>>>>>> index
>>>>>>>>>>>>
>>>>>>>>>>>>  in the
>>>>>>>>>>>>> searchers?
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Greg
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jan 19, 2009 at 3:29 PM, Amin
Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>  wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have recently worked on developing
an application which
>>>>>>>>>>>>>> allows
>>>>>>>>>>>>>> you to
>>>>>>>>>>>>>> upload a file (which is indexed so
you can search later).  I
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>> numerous
>>>>>>>>>>>>>> tests to show that you can index
and search documents (in some
>>>>>>>>>>>>>> instances
>>>>>>>>>>>>>> within the same test), however when
I perform the operation in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> site:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Upload File and Index
>>>>>>>>>>>>>> 2) Search
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't get any hits.  When I restart
the application then if
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  make
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>  another search I can find the results.  It seems as
though
>>>>>>>>>>  indexes
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> aren't
>>>>>>>>>>>>>> being committed when I do the initial
upload.  This is
>>>>>>>>>>>>>>  strange.
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>> explicitly call commit in my code
when I upload the file.  Has
>>>>>>>>>>>>>> anyone
>>>>>>>>>>>>>> experienced this before?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kind Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>  To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Send instant messages to your online friends
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  http://in.messenger.yahoo.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>> Send instant messages to your online friends
>>>>>>> http://in.messenger.yahoo.com
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message