lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandre Rafalovitch <arafa...@gmail.com>
Subject Re: Solr document missing or not getting indexed though we get 200 ok status from server
Date Sun, 04 Sep 2016 16:40:13 GMT
Can you identify the specific documents that 'fail'? What happens if
you post them manually? Try posting them manually but with one field
super-distinct to see whether it made it in. What happens if you post
it to an empty index (copy definition and try).

Also, what's your request handler's parameters look like. Perhaps you
have a signature processor, in which case it may be triggering
duplicates avoidance with different calculation from just an id.

My guess is still that it is some sort of duplicate issue.

Regards,
   Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/


On 4 September 2016 at 23:10, Ganesh M <mganeshs@live.in> wrote:
> Some more information on this... Most of documents get indexed properly. Few documents
are not getting indexed.
>
> All documents POST are seen in the localhost_access and 200 OK response is seen in local
host access file. But in catalina, there are some difference in the logs for which are indexing
properly, following is the logs.
>
> FINE: PRE_UPDATE add
> {,id=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001}
> params(crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001),defaults(wt=xml)
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.TransactionLog <init>
> FINE: New TransactionLog file=/ebdata2/solrdata/IOB_shard1_replica1/data/tlog/tlog.0000000000000220856,
exists=false, size=0, openExisting=false
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.SolrCmdDistributor submit
> FINE: sending update to http://xx.xx.xx.xx:7070/solr/IOB_shard1_replica2/ retry:0 add{version=1544254202941800448,id=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001}
params:update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fxx.xx.xx.xx%3A7070%2Fsolr%2FIOB_shard1_replica1%2F
> Sep 01, 2016 7:39:31 AM org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner
run
> FINE: starting runner: org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner@3fb794b2
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> FINE: PRE_UPDATE FINISH params(crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001),defaults(wt=xml)
> Sep 01, 2016 7:39:31 AM org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner
run
> FINE: finished: org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner@3fb794b2
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [IOB_shard1_replica1] webapp=/solr path=/update params=
> {crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001}
> {add=[CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001
(1544254202941800448)]}
> Sep 01, 2016 7:39:31 AM org.apache.solr.servlet.SolrDispatchFilter doFilter
> FINE: Closing out SolrRequest: params(crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001),defaults(wt=xml)
> -------------------------------------------------
>
> For the one which document is not getting indexed, we could see only following log in
catalina.out. Not sure whether it's getting added to SOLR.
>
>
> Sep 01, 2016 7:39:56 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> FINE: PRE_UPDATE FINISH params(crid=CUA0000004390000019223370564139182810C3LEA0000020179223370567061972057EXCC1000002),defaults(wt=xml)
> Sep 01, 2016 7:39:56 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [IOB_shard1_replica1] webapp=/solr path=/update params=
> {crid=CUA0000004390000019223370564139182810C3LEA0000020179223370567061972057EXCC1000002}
> {} 0 1
> Sep 01, 2016 7:39:56 AM org.apache.solr.servlet.SolrDispatchFilter doFilter
> FINE: Closing out SolrRequest: params(crid=CUA0000004390000019223370564139182810C3LEA0000020179223370567061972057EXCC1000002),defaults(wt=xml)
>
> ----------------------
>
> You can see that in above log for missing documents ( which is not indexed), in catalina
log, we are not seeing "PRE UPDATE ADD". Is that causing / reason for document not getting
indexed ?
>
> We have set autosoftcommit to 1 seconds and autohardcommit to 30 seconds.
>
> We are not getting any errors or exceptions in the log.
>
> This issue is becoming very critical and sort of reliable factor. Though we get 200 OK
response from SOLR for update HTTP POST request, nothing happens on the SOLR side. If SOLR
is not able to process, isn't it we get error from SOLR instead of giving 200 OK response.
>
> Anybody has faced this sort of issue or any sort of help would be very much appreciated.
>
>
>
>
> On Sun, Sep 4, 2016 at 12:59 PM Ganesh M <mganeshs@live.in<mailto:mganeshs@live.in>>
wrote:
> Nitin, Thanks for reply. Our each document has unique id and its hbase rowkey id. So
it will be unique only. So there is no chance of duplicates id being send.
>
>
>
> On Sun 4 Sep, 2016 12:41 pm Nitin Kumar, <nitinkumar.iitm@gmail.com<mailto:nitinkumar.iitm@gmail.com>>
wrote:
> Please check doc's unique key(Id). All keys shd be unique. Else docs having
> same id will be replaced.
>
> On 04-Sep-2016 12:13 PM, "Ganesh M" <mganeshs@live.in<mailto:mganeshs@live.in>>
wrote:
>
>> Hi,
>> we are keep sending documents to Solr from our app server. Single document
>> per request, but in parallel of 10 request hits solr cloud in a second.
>>
>> We could see our post request ( update request ) hitting our solr 5.4 in
>> localhost_access logs, and it's response as 200 Ok response. And also we
>> get HTTP 200 OK response to our app servers as well for out HTTP request we
>> fired to SOLR Cloud.
>>
>> But few documents are not getting indexed. Out of 2000 documents we sent
>> 10 documents are getting missed. Thought there is not error, few documents
>> are getting missed.
>>
>> We use autoSoftcommit as 2 secs and autohardcommit as 30 secs.
>>
>> Why is that 10 documents not getting indexed and also no error getting
>> thrown back if server is not able to index it ?
>>
>> Regards,
>>
>>
>>
>>

Mime
View raw message