lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Wallace>
Subject Strange missing docs when reindexing with threads.
Date Fri, 12 Jun 2009 18:10:23 GMT
Hi all!

I'm using Solr 1.3 and currently testing reindexing...

In my client app, i am sending 17494 requests to add documents...  In 3 
different scenarios:

a) not using threads
b) using 1 thread
c) using 2 threads

In scenario a), everything seems to work fine... In my client log, is 
see 17494 requests sent to solr, in solr's log, I see the same number of 
'add' requests received, and If i search the index, i can see the same 
amount of documents.

However, if I use 1 thread, I see the right amount of requests in logs, 
but I only find 15k or so documents (this varies a bit every time i run 
this scenario).

It gets way worse if I use 2 threads... I can see the right amount of 
requests in both logs, but i end up with ~ 600 docs in the index!

In all scenarios, I don't see any errors on the logs...

As you can imagine, I need to be able to use multiple threads to speed 
up the process... It is also very concertning that I don't get any 
errors anywhere...

Looking at solr's admin stats, I see also 17494 cumulative adds, but 
only a tiny fraction of actual documents can be found...

Any clues?

BTW, these indexers work fine if I use lucene straight...

Thanks in advance for all your help!

View raw message