lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nobody <daily...@mailinator.com>
Subject Re: Spawn an indexing thread on every update
Date Fri, 12 Sep 2008 22:12:03 GMT

Perfect ... However 1 small question ...

every update means finding out what already exists, delete them, then
reindex. which means the following:

Say I have 1000 objects buffered to be indexed later
and I run a thread when i reach the threshold=1000

FOR EACH OBECT:
a. Close IndexWriter
b. Instantiate an IndexReader, delete existing object
c. Instantiate IndexWriter, Index new object
GO BACK TO (a)


Is there a way to avoid some many reinstantiation of objects?

-thanks a ton!





Simon Willnauer wrote:
> 
> I guess similar problems have been discussed on the list over and over
> again.
> Why don't you use a single update thread which takes care of batch
> processing, commit threshold or performs commits after a certain time
> span.
> This would at least prevent you from spawning all the threads which
> is, as far as I can tell not necessary.
> Your update thread could also notify you searcher to reopen the index
> as soon as a change or a batch was commited.
> If you are looking for examples you might find some good examples in
> the Solr code.
> 
> - simon
> 
> On Fri, Sep 12, 2008 at 11:29 AM, Ian Lea <ian.lea@gmail.com> wrote:
>> Hi
>>
>>
>> Why 15 mins?  Can you try lower values to get a balance between load
>> and freshness?
>>
>>
>>
>> --
>> Ian.
>>
>>
>> On Thu, Sep 11, 2008 at 9:43 PM, nobody <dailykos@budweiser.com> wrote:
>>>
>>> Hi,
>>>
>>>  In our application, I want users to be able to search for the updates
>>> they
>>> make almost immediately. Hence, whenever they update, I spawn a thread
>>> immediately to index. However, when the load on the application is very
>>> high
>>> the number of threads spawned increases, and this results in "cannot
>>> create
>>> native thread" error.
>>>
>>> We are thinking of running the indexing thread once in every 15 mins,
>>> through a scheduler, and buffer all the writes till the thread runs.
>>> However, this will result in stale results on the search. Please advice
>>> what
>>> is the best approach in this regard.
>>>
>>> -thanks
>>> --
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Spawn-an-indexing-thread-on-every-update-tp19444244p19465016.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message