lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Walters <greg.walt...@answers.com>
Subject Re: Questions about integrateing SolrCloud with HDFS
Date Thu, 26 Dec 2013 16:33:42 GMT
YouPeng,

While I'm unable to help you with the issue that you're seeing I did want to comment here
and say that I have previously brought up the same goal that you're trying to accomplish on
this mailing list but received no feedback or input. I think it makes sense that Solr should
not try to make its index directories distinct and redundant per shard/core while running
on HDFS as data redundancy and locality is handled at a different layer in the software stack.

+1 to this topic because I'd love to see Solr handle replication/redundancy more smartly on
HDFS

Thanks,
Greg


On Dec 24, 2013, at 1:57 AM, YouPeng Yang <yypvsxf19870706@gmail.com> wrote:

> Hi users
> 
> Solr supports for writing and reading its index and transaction log files
> to the HDFS distributed filesystem.
>  **I am curious about that there are any other futher improvement about
> the integration with HDFS.*
>  **For the solr  native replication  will make multiple copies  of the
> master node's index. Because of the native replication of HDFS,there is no
> need to do that.It just to need that multiple cores in solrcloud share the
> same index directory in HDFS?*
> 
> 
>   The above supposition is what I want to achive when we are integrating
> SolrCloud with HDFS (Solr 4.6).
>   To make sure of our application high available,we still have  to take
> the solr   replication with   some tricks.
> 
>   Firstly ,noting that  solr's index directory is made up of
> *collectionName/coreNodeName/data/index *
> 
> *collectionName/coreNodeName/data/tlog*
>   So to achive this,we want to create multi cores that use the same  hdfs
> index directory .
> 
>  I have tested this  within solr 4.4 by expilcitly indicating  the same
> coreNodeName.
> 
>  For example:
>  Step1, a core was created with the name=core1 and shard=core_shard1 and
> collection=clollection1 and coreNodeName=*core1*
>  Step2. create another core  with the name=core2 and shard=core_shard1 and
> collection=clollection1 and coreNodeName=
> *core1*
> *  T*he two core share the same shard ,collection and coreNodeName.As a
> result,the two core will get the same index data which is stored in the
> hdfs directory :
>  hdfs://myhdfs/*clollection1*/*core1*/data/index
>  hdfs://myhdfs/*clollection1*/*core1*/data/tlog
> 
>  Unfortunately*, *as the solr 4.6 was released,we upgraded . the above
> goal failed. We could not create a core with both expilcit shard and
> coreNodeName.
> Exceptions are as [1].
> *  Can some give some help?*
> 
> 
> Regards
> [1]------------------------------------------------------------------------------------------------------------------
> 64893635 [http-bio-8080-exec-1] INFO  org.apache.solr.cloud.ZkController
> ?.publishing core=hdfstest3 state=down
> 64893635 [http-bio-8080-exec-1] INFO  org.apache.solr.cloud.ZkController
> ?.numShards not found on descriptor - reading it from system property
> 64893698 [http-bio-8080-exec-1] INFO  org.apache.solr.cloud.ZkController
> ?.look for our core node name
> 
> 
> 
> 64951227 [http-bio-8080-exec-17] INFO  org.apache.solr.core.SolrCore
> ?.[reportCore_201208] webapp=/solr path=/replication
> params={slave=false&command=details&wt=javabin&qt=/replication&version=2}
> status=0 QTime=107
> 
> 
> 65213770 [http-bio-8080-exec-1] INFO  org.apache.solr.cloud.ZkController
> ?.waiting to find shard id in clusterstate for hdfstest3
> 65533894 [http-bio-8080-exec-1] ERROR org.apache.solr.core.SolrCore
> ?.org.apache.solr.common.SolrException: Error CREATEing SolrCore
> 'hdfstest3': Could not get shard id for core: hdfstest3
>    at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:535)
>    at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
>    at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
>    at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
>    at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>    at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>    at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>    at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>    at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>    at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>    at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
>    at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>    at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>    at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
>    at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>    at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>    at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>    at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.solr.common.SolrException: Could not get shard id for
> core: hdfstest3
>    at
> org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1302)
>    at
> org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1248)
>    at
> org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1399)
>    at
> org.apache.solr.core.CoreContainer.preRegisterInZk(CoreContainer.java:942)
>    at
> org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:492)
>    ... 20 more


Mime
View raw message