lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JohnWu (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (SOLR-1395) Integrate Katta
Date Fri, 30 Dec 2011 06:49:36 GMT

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139997#comment-13139997
] 

JohnWu edited comment on SOLR-1395 at 12/30/11 6:49 AM:
--------------------------------------------------------

tom:
   For multi-schema case, I try to add multi-core in proxy and sub-proxy, but I meet the connection
close issue.
   please help me and analyze the case.
   
   now in proxy, I set the solr home with proxymulticore folder, which contains structure
as follows:
   -proxymulticore
   --customer
   ----conf
   ----data
   --part
   ----conf
   ----data
   --solr.xml

  solr.xml as follows:
  <cores adminPath="/admin/cores">
	<core name="part" instanceDir="part">		
	</core>
	<core name="customer" instanceDir="customer">		
	</core>
  </cores>

  in part folder the solrcong.xml set as 
----------------
  <requestHandler name="standard" class="solr.KattaRequestHandler" default="true">
    <lst name="defaults">
        <str name="echoParams">explicit</str>
        <str name="shards">part-00000,part-00001,part-00002</str>
     </lst>
  </requestHandler>
----------------

  so in the proxy, we can use the request:
http://localhost:8080/solr-1395-katta-0.6.2-2patch/part/select/?q=a*&version=2.2&start=0&rows=10&indent=on&isShard=false&distrib=true&core=part

  the proxy will use the kattaRequest dispatch the query to katta datanodes(subproxy).

  in subproxy, when we start the embedded solr with subproxymulticore folder, which structure
as follows:
   -subproxymulticore
   --customer
   ----conf
   ----data
   --part
   ----conf
   ----data
   --solr.xml

   exlusive the solrconfig.xml in part folder as follows, the others are same.
----------------
  <requestHandler name="standard" class="solr.SearchHandler" default="true">
    <!-- default values for query parameters -->
     <lst name="defaults">
       <str name="echoParams">explicit</str>
     </lst>
  </requestHandler>
------------------ 
  now, I correct some code of DeployableSolrKattaServer.java in patched solr as follows:
--------------------
	public DeployableSolrKattaServer() throws ParserConfigurationException,
			IOException, SAXException {
//		super(getServerName(), new CoreContainer(getSolrHome()
//				.getAbsolutePath(), getConfigFile()));
		//super(getServerName(), new CoreContainer());

//By JohnWu, we do not direct find the conf folder of a core, we find the solr.xml to add
cores.

		super(getServerName(),new CoreContainer.Initializer().initialize());
		

	}

-------------------

add correct some code in SolrKattaServer.java

-------------------
	public SolrKattaServer(String defaultCoreName, CoreContainer coreContainer) {
		this.coreContainer = coreContainer;
		handler = new MultiEmbeddedSearchHandler(coreContainer);
		handler.init(new NamedList());
		//defaultCore = coreContainer.getCore(defaultCoreName);
		defaultCores = coreContainer.getCores();
//		if (defaultCore == null)
//			throw new SolrException(ErrorCode.UNKNOWN, "defaultCore:"
//					+ defaultCoreName + " could not be found");
		if (defaultCores == null)
			throw new SolrException(ErrorCode.UNKNOWN, "defaultCore:"
					+ defaultCoreName + " could not be found");
		//JohnWu add for multi-cores
		Iterator it = defaultCores.iterator();
		while(it.hasNext()){
			handler.inform((SolrCore)it.next());
		}
		//handler.inform(defaultCore);
	}


	/**
	 * The main method that executes requests from a KattaClient
	 */
	@Override
	public KattaResponse request(String[] shards, KattaRequest request)
			throws Exception {
		
		//JohnWu add it for multi cores, we need get the suitable core according to the shard
	    SolrParams params = request.getParams();
	    SolrParams required = params.required();
	    String cname = required.get(CoreAdminParams.CORE);
	    SolrCore core = coreContainer.getCore(cname);
	    
	    //need add some code to void the socre is null
	    if (core != null) {
			ModifiableSolrParams sp = new ModifiableSolrParams(request.getParams());
			String shardsStr = StringUtils.arrayToString(shards);
			sp.set(ShardParams.SHARDS, shardsStr);

			if (log.isDebugEnabled()){
				log.debug("SolrServer.request: " + nodeName + " shards:"
						+ Arrays.asList(shards) + " request params:" + sp );
			}

			//remove by John
			//SolrQueryRequestBase req = new LocalSolrQueryRequest(defaultCore, sp);
			

				SolrQueryRequestBase req = new LocalSolrQueryRequest(core, sp);
				SolrQueryResponse resp = new SolrQueryResponse();
				// Added by tom liu
				// because exception would stop RPC
				// so, must handle exception
				try{
				// add end
					getRequestHandler(req).handleRequest(req, resp);
				// Added by tom liu
				}catch(SolrException ex){
					log.error(ex.getMessage(), ex);
				}
				// add end
				NamedList nl = resp.getValues();
				nl.add("QueriedShards", shards);
				// Added by tom liu
				SolrDocumentList sdl = (SolrDocumentList)nl.get("response");
				if( sdl == null ){
					nl.add("response", new SolrDocumentList());
					if( log.isWarnEnabled() )
						log.warn("SolrServer.SolrResponse: no response");
				}
				// add end
				SolrResponse rsp = new SolrResponseBase();
				rsp.setResponse(nl);
				if (log.isDebugEnabled()){
					if( null != sdl ){
						log.debug("SolrServer.SolrResponse: numFound=" + sdl.getNumFound() 
								+ ",start=" + sdl.getStart() + ",docs=" + sdl.size());
					}
					log.debug("termVectors=" + nl.get("termVectors"));
				}
// By using shards[0] we guarantee that this response is tied to a known
// shard in the orignator, so that the results can be merged.
// The name and only 1 is allowed has to be one of the original query
// shards.

				return new KattaResponse(shards[0], "", 0, rsp);
	    }else{
	    	//maybe null is bad!
	    	System.out.println("------the core is null!!!!!!");
	    	return null;
	    }					
		
		

	}

	// Added by tom liu
	// for supporting qt=...
	private MultiEmbeddedSearchHandler getRequestHandler(SolrQueryRequest request) {
	    SolrParams params = request.getParams();
	    if( params == null ) {
	      params = new ModifiableSolrParams();
	    }

	    String qt = params.get( CommonParams.QT );
	    if (qt != null) {	
	    	//JohnWu remove follow for multi-core
//	    	MultiEmbeddedSearchHandler myhandler = (MultiEmbeddedSearchHandler)defaultCore.getRequestHandler(
qt );
//	        if( myhandler == null ) {
//	          throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, "unknown handler:
"+qt);
//	        }
//	        myhandler.setCoreContainer(coreContainer);
//	        return myhandler;
	        
	    	
	    	//JohnWu add for multi-core
	    	
			Iterator it = defaultCores.iterator();
			while(it.hasNext()){
		    	MultiEmbeddedSearchHandler myhandler = (MultiEmbeddedSearchHandler)((SolrCore)it.next()).getRequestHandler(
qt );
		        if( myhandler == null ) {
		          throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, "unknown handler:
"+qt);
		        }
		        myhandler.setCoreContainer(coreContainer);
		        return myhandler;
			}

	    }
		return handler;
	}

------------------

but now when the katta send the second query with ids to data node, the connection is closed.

INFO: [pSearch-00000#pSearch-00000] webapp=null path=/select params={start=0&ids=28aaa%2Caaa%2Cbbb&q=a*&core=part&isShard=true&rows=10}
hits=3 status=0 QTime=12

please help me, and tell me why the process can not go through the whole process.

Thanks!

JohnWu


   
  

                
      was (Author: johnwu):
    tom:
   For multi-schema case, I try to add multi-core in proxy and sub-proxy, but I meet the connection
close issue.
   please help me and analyze the case.
   
   now in proxy, I set the solr home with proxymulticore folder, which contains structure
as follows:
   -proxymulticore
   --customer
   ----conf
   ----data
   --part
   ----conf
   ----data
   --solr.xml

  solr.xml as follows:
  <cores adminPath="/admin/cores">
	<core name="part" instanceDir="part">		
	</core>
	<core name="customer" instanceDir="customer">		
	</core>
  </cores>

  in part folder the solrcong.xml set as 
----------------
  <requestHandler name="standard" class="solr.KattaRequestHandler" default="true">
    <lst name="defaults">
        <str name="echoParams">explicit</str>
        <str name="shards">part-00000,part-00001,part-00002</str>
     </lst>
  </requestHandler>
----------------

  so in the proxy, we can use the request:
http://localhost:8080/solr-1395-katta-0.6.2-2patch/part/select/?q=a*&version=2.2&start=0&rows=10&indent=on&isShard=false&distrib=true&core=part

  the proxy will use the kattaRequest dispatch the query to katta datanodes(subproxy).

  in subproxy, when we start the embedded solr with subproxymulticore folder, which structure
as follows:
   -subproxymulticore
   --customer
   ----conf
   ----data
   --part
   ----conf
   ----data
   --solr.xml

   exlusive the solrconfig.xml in part folder as follows, the others are same.
----------------
  <requestHandler name="standard" class="solr.SearchHandler" default="true">
    <!-- default values for query parameters -->
     <lst name="defaults">
       <str name="echoParams">explicit</str>
     </lst>
  </requestHandler>
------------------ 
  now, I correct some code of DeployableSolrKattaServer.java in patched solr as follows:
--------------------
	public DeployableSolrKattaServer() throws ParserConfigurationException,
			IOException, SAXException {
//		super(getServerName(), new CoreContainer(getSolrHome()
//				.getAbsolutePath(), getConfigFile()));
		//super(getServerName(), new CoreContainer());

//By JohnWu, we do not direct find the conf folder of a core, we find the solr.xml to add
cores.

		super(getServerName(),new CoreContainer.Initializer().initialize());
		

	}

-------------------

add correct some code in SolrKattaServer.java

-------------------
	public SolrKattaServer(String defaultCoreName, CoreContainer coreContainer) {
		this.coreContainer = coreContainer;
		handler = new MultiEmbeddedSearchHandler(coreContainer);
		handler.init(new NamedList());
		//defaultCore = coreContainer.getCore(defaultCoreName);
		defaultCores = coreContainer.getCores();
//		if (defaultCore == null)
//			throw new SolrException(ErrorCode.UNKNOWN, "defaultCore:"
//					+ defaultCoreName + " could not be found");
		if (defaultCores == null)
			throw new SolrException(ErrorCode.UNKNOWN, "defaultCore:"
					+ defaultCoreName + " could not be found");
		//JohnWu add for multi-cores
		Iterator it = defaultCores.iterator();
		while(it.hasNext()){
			handler.inform((SolrCore)it.next());
		}
		//handler.inform(defaultCore);
	}


	/**
	 * The main method that executes requests from a KattaClient
	 */
	@Override
	public KattaResponse request(String[] shards, KattaRequest request)
			throws Exception {
		
		//JohnWu add it for multi cores, we need get the suitable core according to the shard
	    SolrParams params = request.getParams();
	    SolrParams required = params.required();
	    String cname = required.get(CoreAdminParams.CORE);
	    SolrCore core = coreContainer.getCore(cname);
	    
	    //need add some code to void the socre is null
	    if (core != null) {
			ModifiableSolrParams sp = new ModifiableSolrParams(request.getParams());
			String shardsStr = StringUtils.arrayToString(shards);
			sp.set(ShardParams.SHARDS, shardsStr);

			if (log.isDebugEnabled()){
				log.debug("SolrServer.request: " + nodeName + " shards:"
						+ Arrays.asList(shards) + " request params:" + sp );
			}

			//remove by John
			//SolrQueryRequestBase req = new LocalSolrQueryRequest(defaultCore, sp);
			

				SolrQueryRequestBase req = new LocalSolrQueryRequest(core, sp);
				SolrQueryResponse resp = new SolrQueryResponse();
				// Added by tom liu
				// because exception would stop RPC
				// so, must handle exception
				try{
				// add end
					getRequestHandler(req).handleRequest(req, resp);
				// Added by tom liu
				}catch(SolrException ex){
					log.error(ex.getMessage(), ex);
				}
				// add end
				NamedList nl = resp.getValues();
				nl.add("QueriedShards", shards);
				// Added by tom liu
				SolrDocumentList sdl = (SolrDocumentList)nl.get("response");
				if( sdl == null ){
					nl.add("response", new SolrDocumentList());
					if( log.isWarnEnabled() )
						log.warn("SolrServer.SolrResponse: no response");
				}
				// add end
				SolrResponse rsp = new SolrResponseBase();
				rsp.setResponse(nl);
				if (log.isDebugEnabled()){
					if( null != sdl ){
						log.debug("SolrServer.SolrResponse: numFound=" + sdl.getNumFound() 
								+ ",start=" + sdl.getStart() + ",docs=" + sdl.size());
					}
					log.debug("termVectors=" + nl.get("termVectors"));
				}
// By using shards[0] we guarantee that this response is tied to a known
// shard in the orignator, so that the results can be merged.
// The name and only 1 is allowed has to be one of the original query
// shards.

				return new KattaResponse(shards[0], "", 0, rsp);
	    }else{
	    	//maybe null is bad!
	    	System.out.println("------the core is null!!!!!!");
	    	return null;
	    }					
		
		

	}

	// Added by tom liu
	// for supporting qt=...
	private MultiEmbeddedSearchHandler getRequestHandler(SolrQueryRequest request) {
	    SolrParams params = request.getParams();
	    if( params == null ) {
	      params = new ModifiableSolrParams();
	    }

	    String qt = params.get( CommonParams.QT );
	    if (qt != null) {	
	    	//JohnWu remove follow for multi-core
//	    	MultiEmbeddedSearchHandler myhandler = (MultiEmbeddedSearchHandler)defaultCore.getRequestHandler(
qt );
//	        if( myhandler == null ) {
//	          throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, "unknown handler:
"+qt);
//	        }
//	        myhandler.setCoreContainer(coreContainer);
//	        return myhandler;
	        
	    	
	    	//JohnWu add for multi-core
	    	
			Iterator it = defaultCores.iterator();
			while(it.hasNext()){
		    	MultiEmbeddedSearchHandler myhandler = (MultiEmbeddedSearchHandler)((SolrCore)it.next()).getRequestHandler(
qt );
		        if( myhandler == null ) {
		          throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, "unknown handler:
"+qt);
		        }
		        myhandler.setCoreContainer(coreContainer);
		        return myhandler;
			}

	    }
		return handler;
	}

------------------

but now when the katta send the second query with ids to data node, the connection is closed.

INFO: [partSearch-00000#partSearch-00000] webapp=null path=/select params={start=0&ids=28aaa%2Caaa%2Cbbb&q=a*&core=part&isShard=true&rows=10}
hits=3 status=0 QTime=12

please help me, and tell me why the process can not go through the whole process.

Thanks!

JohnWu


   
  

                  
> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: 3.6, 4.0
>
>         Attachments: SOLR-1395.patch, SOLR-1395.patch, SOLR-1395.patch, back-end.log,
front-end.log, hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar, katta-solrcores.jpg, katta.node.properties,
katta.zk.properties, log4j-1.2.13.jar, solr-1395-1431-3.patch, solr-1395-1431-4.patch, solr-1395-1431-katta0.6.patch,
solr-1395-1431-katta0.6.patch, solr-1395-1431.patch, solr-1395-katta-0.6.2-1.patch, solr-1395-katta-0.6.2-2.patch,
solr-1395-katta-0.6.2-3.patch, solr-1395-katta-0.6.2.patch, solr-1395-katta-0.6.3-4.patch,
solr-1395-katta-0.6.3-5.patch, solr-1395-katta-0.6.3-6.patch, solr-1395-katta-0.6.3-7.patch,
solr1395.jpg, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message