lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Macroman <peter0...@hotmail.com>
Subject Re: Maximum Number of Records In Index
Date Thu, 14 Feb 2013 13:27:52 GMT
Partial updates is nothing as clever as I may have made it sound, it is just
changing a record value , for example last name from Smith to Jones, that's
my partial update. 

No errors at all in indexing, I have not yet checked the logs , but the DIH
output counts show no errors, here is an example
<str name="Total Requests made to DataSource">2</str><str name="Total Rows
Fetched">14823</str><str name="Total Documents Skipped">0</str><str
name="Full Dump Started">2013-02-14 07:00:30</str><str name="">Indexing
completed. Added/Updated: 14823 documents. Deleted 0 documents.</str><str
name="Committed">2013-02-14 07:19:59</str><str name="Optimized">2013-02-14
07:19:59</str><str name="Total Documents Processed">14823</str><str
name="Time taken ">0:19:58.557</str>

Having analysed the SOLR index this afternoon I realised that I actually add
the date/time of when record indexed so did a quick SOLR admin count using
.... record_date:[2000-02-14T00:00:00.000Z TO 2013-02-10T00:00:00.000Z]....
this resulted in a count of 32.723 records indexed today, and when I add up
all the DIH's of Added/Updated it comes to 35,369 , weird !!! Now for the
total maths , yesterday's total index count was 13593885 and today it is
13598211 a difference of 4326, but I do need to take into account records
updates, so running the SQL form each of the DIH's sources in SQL Developer
to purely get counts, my counts are a total of 31,789 which means only 3,000
to 4,000 updates the rest are all new.

So I will definately say that records are being deleted so need to check the
logs as suggested. If no mention of deletions exist my next question will be
can I get a Month-YYYY breakdown on a SOLR date field so I can monitor
records that drop off, because one field that will definately not change is
the record creation date from the source systems which is part of the
indexed record?

<this line ready for entering log details to see if any deletes occurred>



--
View this message in context: http://lucene.472066.n3.nabble.com/Maximum-Number-of-Records-In-Index-tp4038961p4040445.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message