lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shawn Heisey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-9466) During concurrency some Solr document are not seen even after soft and hard commit
Date Thu, 01 Sep 2016 13:23:21 GMT

    [ https://issues.apache.org/jira/browse/SOLR-9466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15455370#comment-15455370
] 

Shawn Heisey commented on SOLR-9466:
------------------------------------

Situations like this should be brought up on the mailing list or the IRC channel before being
opened in the bugtracker, so we can determine whether they are actually bugs.

For your problem, which I do not believe is a bug:

You have told Solr to soft commit after one second, but this doesn't mean that the commit
will actually *complete* within one second -- only that it will *start* within one second.
 I've seen commits take a minute or more.  Usually this is a misconfiguration, where cache
autowarmCount values are too high, or there's not enough memory available.

https://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits

Another possible problem when using a third-party container like tomcat is that the container's
maxThreads setting is too low.  This setting defaults to 200, but in the Jetty that comes
with Solr, it is set to 10000, so that Solr can create as many threads as it needs.  With
high concurrency, the number of threads required can easily exceed 200.

In the unlikely situation that this is a bug, we would need to see the bug demonstrated in
a 6.x version of Solr.  Bugs in 4.x are not going to be fixed, unless it's a MAJOR showstopper
bug.


> During concurrency some Solr document are not seen even after soft and hard commit
> ----------------------------------------------------------------------------------
>
>                 Key: SOLR-9466
>                 URL: https://issues.apache.org/jira/browse/SOLR-9466
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>    Affects Versions: 4.10.2
>         Environment: Cent OS
>            Reporter: Ganesh
>            Priority: Critical
>
> Solr cloud with 2 nodes, master master, with 5 collection and 2 shards in each collection.

> During concurrent usage of SOLR where both updates and search is sent to SOLR server,
some of our updates / adding of new documents are getting lost.
> We could see that update hitting solr and we could see it in localhost_access file of
tomcat, also in catalina.out. But still we couldn't see that record while searching. 
> Following is the catalina.out logs for the document which is getting indexed properly.
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.processor.LogUpdateProcessor processAdd
> FINE: PRE_UPDATE add{,id=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001}
{{params(crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001),defaults(wt=xml)}}
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.TransactionLog <init>
> FINE: New TransactionLog file=/ebdata2/solrdata/IOB_shard1_replica1/data/tlog/tlog.0000000000000220856,
exists=false, size=0, openExisting=false
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.SolrCmdDistributor submit
> FINE: sending update to http://xx.xx.xx.xx:7070/solr/IOB_shard1_replica2/ retry:0 add{_version_=1544254202941800448,id=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001}
params:update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fxx.xx.xx.xx%3A7070%2Fsolr%2FIOB_shard1_replica1%2F
> Sep 01, 2016 7:39:31 AM org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner
run
> FINE: starting runner: org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner@3fb794b2
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> FINE: PRE_UPDATE FINISH {{params(crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001),defaults(wt=xml)}}
> Sep 01, 2016 7:39:31 AM org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner
run
> FINE: finished: org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner@3fb794b2
> Sep 01, 2016 7:39:31 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [IOB_shard1_replica1] webapp=/solr path=/update params={crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001}
{add=[CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001 (1544254202941800448)]}
0 9
> Sep 01, 2016 7:39:31 AM org.apache.solr.servlet.SolrDispatchFilter doFilter
> FINE: Closing out SolrRequest: {{params(crid=CUA0000004390000019223370564139207241C3LEA0000020769223370567404392838EXCC3000001),defaults(wt=xml)}}
> For the one which document is not getting indexed, we could see only following log in
catalina.out. Not sure whether it's getting added to SOLR.
> Sep 01, 2016 7:39:56 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> FINE: PRE_UPDATE FINISH {{params(crid=CUA0000004390000019223370564139182810C3LEA0000020179223370567061972057EXCC1000002),defaults(wt=xml)}}
> Sep 01, 2016 7:39:56 AM org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: [IOB_shard1_replica1] webapp=/solr path=/update params={crid=CUA0000004390000019223370564139182810C3LEA0000020179223370567061972057EXCC1000002}
{} 0 1
> Sep 01, 2016 7:39:56 AM org.apache.solr.servlet.SolrDispatchFilter doFilter
> FINE: Closing out SolrRequest: {{params(crid=CUA0000004390000019223370564139182810C3LEA0000020179223370567061972057EXCC1000002),defaults(wt=xml)}}
> We have set autosoftcommit to 1 seconds and autohardcommit to 30 seconds.
> We are not getting any errors or exceptions in the log. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message