lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer" <simon.willna...@googlemail.com>
Subject Re: Spawn an indexing thread on every update
Date Fri, 12 Sep 2008 23:02:37 GMT
you can just simply do it like the following:

  open new IndexReader
  delete all docs
  close IndexReader
  open new IndexWriter
  add documents
  close IndexWriter

if you are done you can replace the index reader used for searching
with a new one an all modifications will be visible.

- simon
On Sat, Sep 13, 2008 at 12:12 AM, nobody <dailykos@mailinator.com> wrote:
>
> Perfect ... However 1 small question ...
>
> every update means finding out what already exists, delete them, then
> reindex. which means the following:
>
> Say I have 1000 objects buffered to be indexed later
> and I run a thread when i reach the threshold=1000
>
> FOR EACH OBECT:
> a. Close IndexWriter
> b. Instantiate an IndexReader, delete existing object
> c. Instantiate IndexWriter, Index new object
> GO BACK TO (a)
>
>
> Is there a way to avoid some many reinstantiation of objects?
>
> -thanks a ton!
>
>
>
>
>
> Simon Willnauer wrote:
>>
>> I guess similar problems have been discussed on the list over and over
>> again.
>> Why don't you use a single update thread which takes care of batch
>> processing, commit threshold or performs commits after a certain time
>> span.
>> This would at least prevent you from spawning all the threads which
>> is, as far as I can tell not necessary.
>> Your update thread could also notify you searcher to reopen the index
>> as soon as a change or a batch was commited.
>> If you are looking for examples you might find some good examples in
>> the Solr code.
>>
>> - simon
>>
>> On Fri, Sep 12, 2008 at 11:29 AM, Ian Lea <ian.lea@gmail.com> wrote:
>>> Hi
>>>
>>>
>>> Why 15 mins?  Can you try lower values to get a balance between load
>>> and freshness?
>>>
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> On Thu, Sep 11, 2008 at 9:43 PM, nobody <dailykos@budweiser.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>>  In our application, I want users to be able to search for the updates
>>>> they
>>>> make almost immediately. Hence, whenever they update, I spawn a thread
>>>> immediately to index. However, when the load on the application is very
>>>> high
>>>> the number of threads spawned increases, and this results in "cannot
>>>> create
>>>> native thread" error.
>>>>
>>>> We are thinking of running the indexing thread once in every 15 mins,
>>>> through a scheduler, and buffer all the writes till the thread runs.
>>>> However, this will result in stale results on the search. Please advice
>>>> what
>>>> is the best approach in this regard.
>>>>
>>>> -thanks
>>>> --
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Spawn-an-indexing-thread-on-every-update-tp19444244p19465016.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message