lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr4.4 or zookeeper 3.4.5 do not support too many collections? more than 600?
Date Sat, 07 Sep 2013 18:14:56 GMT
Right, I _think_ that the use of ZK is limited to 1M and it looks like the
600th collection pushes the ZK state past 1M. 1024*1024 is
1,048,576 which is waaaaay suspiciously close to
1,048,971

At 600 collections you're pushing past this limit it looks like.
Not quite sure where it can be  changed. Here's a good discussion of this:
http://lucene.472066.n3.nabble.com/gt-1MB-file-to-Zookeeper-td3958614.html

Best,
Erick


On Sat, Sep 7, 2013 at 10:10 AM, diyun2008 <diyun2008@gmail.com> wrote:

> *I have installed solr cloud with solr4.4 and zookeeper 3.4.5.
> And I'm testing some requirements with 10k collections supporting in one
> solr server.
> When I post collection to solr
>
> server(admin/collections?action=CREATE&name=europetest${loopcnt}&numShards=2&replicationFactor=2&maxShardsPerNode=2)
> with jmeter,
> I found every time when collections number reached 600+, Solr and zookeeper
> will not work correctly.
>
> I checked logs. Here's Solr logs:*
> 07:56:01,149 ERROR SolrException:
> null:org.apache.solr.common.SolrException:
> createcollection the collection time out:60s
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:175)
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:156)
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:290)
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:112)
>         at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>         at
>
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
>         at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
>         at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>         at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>         at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>         at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>         at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>         at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>         at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>         at
>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
>         at
>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>         at
>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:781)
>
> 07:57:23,523 ERROR SolrException: org.apache.solr.common.SolrException:
> createcollection the collection error [Watcher fired on path:
> /overseer/collection-queue-work/qnr-0000001590 state: SyncConnected type
> NodeDeleted]
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:178)
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:156)
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleCreateAction(CollectionsHandler.java:290)
>         at
>
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:112)
>         at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>         at
>
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611)
>         at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
>         at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>         at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>         at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>         at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>         at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>         at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>         at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>         at
>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
>         at
>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>         at
>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:781)
>
>
> And here's the zookeeper logs:
>
> 2013-09-07 15:56:17,498 [myid:1] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception
> causing close of session 0x140f624fa990000 due to java.io.IOException: Len
> error 1048971
> 2013-09-07 15:56:17,507 [myid:1] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed
> socket connection for client /9.110.83.131:21210 which had sessionid
> 0x140f624fa990000
> 2013-09-07 15:56:29,966 [myid:1] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
> Accepted socket connection from /9.110.83.131:23793
> 2013-09-07 15:56:29,967 [myid:1] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client
> attempting to renew session 0x140f624fa990000 at /9.110.83.131:23793
> 2013-09-07 15:56:29,967 [myid:1] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@107] - Revalidating
> client: 0x140f624fa990000
> 2013-09-07 15:56:29,968 [myid:1] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@595] -
> Established
> session 0x140f624fa990000 with negotiated timeout 120000 for client
> /9.110.83.131:23793
> 2013-09-07 15:56:51,124 [myid:1] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception
> causing close of session 0x140f624fa990000 due to java.io.IOException: Len
> error 1048971
> 2013-09-07 15:56:51,125 [myid:1] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed
> socket connection for client /9.110.83.131:23793 which had sessionid
> 0x140f624fa990000
>
> *I'm very confused by this problem. I have tried add more memory to
> zookeeper and solr. but it's useless.
> I'm very appreciated if someone can help me.*
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr4-4-or-zookeeper-3-4-5-do-not-support-too-many-collections-more-than-600-tp4088689.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message