lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Wallace ...@rwmotloc.com>
Subject Re: Strange missing docs when reindexing with threads.
Date Fri, 12 Jun 2009 22:24:04 GMT
That was exactly my issue... i changed my code to not reuse 
document/fields and it is all good now!

Thanks for your support!

Shalin Shekhar Mangar wrote:
> On Fri, Jun 12, 2009 at 11:40 PM, Alexander Wallace <aw@rwmotloc.com> wrote:
>
>   
>> Hi all!
>>
>> I'm using Solr 1.3 and currently testing reindexing...
>>
>> In my client app, i am sending 17494 requests to add documents...  In 3
>> different scenarios:
>>
>> a) not using threads
>> b) using 1 thread
>> c) using 2 threads
>>
>> In scenario a), everything seems to work fine... In my client log, is see
>> 17494 requests sent to solr, in solr's log, I see the same number of 'add'
>> requests received, and If i search the index, i can see the same amount of
>> documents.
>>
>> However, if I use 1 thread, I see the right amount of requests in logs, but
>> I only find 15k or so documents (this varies a bit every time i run this
>> scenario).
>>
>> It gets way worse if I use 2 threads... I can see the right amount of
>> requests in both logs, but i end up with ~ 600 docs in the index!
>>
>> In all scenarios, I don't see any errors on the logs...
>>
>> As you can imagine, I need to be able to use multiple threads to speed up
>> the process... It is also very concertning that I don't get any errors
>> anywhere...
>>
>> Looking at solr's admin stats, I see also 17494 cumulative adds, but only a
>> tiny fraction of actual documents can be found...
>>
>> Any clues?
>>
>>     
>
> What is the uniqueKey in your schema.xml? Is it possible that those 17494
> documents have a common uniqueKey and are therefore getting overwritten?
>
>   

Mime
View raw message