lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tom liu (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (SOLR-1395) Integrate Katta
Date Sun, 10 Oct 2010 07:28:31 GMT

    [ https://issues.apache.org/jira/browse/SOLR-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919478#action_12919478
] 

tom liu edited comment on SOLR-1395 at 10/10/10 3:26 AM:
---------------------------------------------------------

i use solr-4.0 newest code trunk, and katta 0.6.2, hadoop-0.20.2, zookeeper-3.3.1, after fixed
some bugs , i run it.

the bugs is :
1. solr's ShardDoc.java, ShardFieldSortedHitQueue line 210 :
<pre>
        final float f1 = e1.score==null?0.00f:e1.score;
        final float f2 = e2.score==null?0.00f:e2.score;
</pre>
2. KattaSearchHandler.java, KattaMultiShardHandler may be return more results, so must include
any results:
<pre>
			if (results.isEmpty()) {
				ssr.setResponse(new NamedList<Object>());
				return;
			}
+
+			NamedList<Object> nl = new NamedList<Object>();
+			NamedListCollection nlc = new NamedListCollection(nl);
+			for(KattaResponse kr : results){
+				nl = nlc.add(kr.getRsp().getResponse());
+			}
			ssr.setResponse(nl);
                }
+		private class NamedListCollection {
+			private NamedList<Object> _nl;
+			NamedListCollection(NamedList<Object> nl){
+				_nl = nl;
+			}
+			NamedList<Object> add(NamedList<Object> nl){
+				Iterator<Entry<String,Object>> it = nl.iterator();
+				while (it.hasNext()){
+					Entry<String,Object> entry = it.next();
+					String key = entry.getKey();
+					Object obj = entry.getValue();
+					Object old = _nl.remove(key);
+					if(old != null){
+						add(key, obj , old );
+					}else{
+						_nl.add(key, obj);
+					}
+				}
+				return _nl;
+			}
+			void add(String key,Object obj,Object old){
+				if(key.equals("response")){
+					SolrDocumentList doca = (SolrDocumentList)obj;
+					SolrDocumentList docb = (SolrDocumentList)old;
+					SolrDocumentList docs = new SolrDocumentList();
+					docs.setNumFound(doca.getNumFound()+docb.getNumFound());
+					//doca.setStart(doca.getStart()+docb.getStart());
+					docs.setMaxScore(Math.max(doca.getMaxScore(), docb.getMaxScore()));
+					docs.addAll(doca);
+					docs.addAll(docb);
+					_nl.add(key,docs);
+				}else if(key.equals("QueriedShards")){
+					Collection<String> qsa = (ArrayList<String>)obj;
+					Collection<String> qsb = (ArrayList<String>)old;
+					Collection<String> qs = new ArrayList<String>();
+					qs.addAll(qsa);
+					qs.addAll(qsb);
+					_nl.add(key, qs);
+				}
+			}
+		}
</pre>

      was (Author: tom_lt):
    i use solr-4.0 newest code trunk, and katta 0.6.2, hadoop-0.20.2, zookeeper-3.3.1, after
fixed some bugs , i run it.

the bugs is :
1. solr's ShardDoc.java, ShardFieldSortedHitQueue line 210 :
        final float f1 = e1.score==null?0.00f:e1.score;
        final float f2 = e2.score==null?0.00f:e2.score;
2. KattaSearchHandler.java, KattaMultiShardHandler may be return more results, so must include
any results:
			if (results.isEmpty()) {
				ssr.setResponse(new NamedList<Object>());
				return;
			}
+
+			NamedList<Object> nl = new NamedList<Object>();
+			NamedListCollection nlc = new NamedListCollection(nl);
+			for(KattaResponse kr : results){
+				nl = nlc.add(kr.getRsp().getResponse());
+			}
			ssr.setResponse(nl);
                }
+		private class NamedListCollection {
+			private NamedList<Object> _nl;
+			NamedListCollection(NamedList<Object> nl){
+				_nl = nl;
+			}
+			NamedList<Object> add(NamedList<Object> nl){
+				Iterator<Entry<String,Object>> it = nl.iterator();
+				while (it.hasNext()){
+					Entry<String,Object> entry = it.next();
+					String key = entry.getKey();
+					Object obj = entry.getValue();
+					Object old = _nl.remove(key);
+					if(old != null){
+						add(key, obj , old );
+					}else{
+						_nl.add(key, obj);
+					}
+				}
+				return _nl;
+			}
+			void add(String key,Object obj,Object old){
+				if(key.equals("response")){
+					SolrDocumentList doca = (SolrDocumentList)obj;
+					SolrDocumentList docb = (SolrDocumentList)old;
+					SolrDocumentList docs = new SolrDocumentList();
+					docs.setNumFound(doca.getNumFound()+docb.getNumFound());
+					//doca.setStart(doca.getStart()+docb.getStart());
+					docs.setMaxScore(Math.max(doca.getMaxScore(), docb.getMaxScore()));
+					docs.addAll(doca);
+					docs.addAll(docb);
+					_nl.add(key,docs);
+				}else if(key.equals("QueriedShards")){
+					Collection<String> qsa = (ArrayList<String>)obj;
+					Collection<String> qsb = (ArrayList<String>)old;
+					Collection<String> qs = new ArrayList<String>();
+					qs.addAll(qsa);
+					qs.addAll(qsb);
+					_nl.add(key, qs);
+				}
+			}
+		}

  
> Integrate Katta
> ---------------
>
>                 Key: SOLR-1395
>                 URL: https://issues.apache.org/jira/browse/SOLR-1395
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>             Fix For: Next
>
>         Attachments: back-end.log, front-end.log, hadoop-core-0.19.0.jar, katta-core-0.6-dev.jar,
katta.node.properties, katta.zk.properties, log4j-1.2.13.jar, solr-1395-1431-3.patch, solr-1395-1431-4.patch,
solr-1395-1431-katta0.6.patch, solr-1395-1431-katta0.6.patch, solr-1395-1431.patch, SOLR-1395.patch,
SOLR-1395.patch, SOLR-1395.patch, test-katta-core-0.6-dev.jar, zkclient-0.1-dev.jar, zookeeper-3.2.1.jar
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> We'll integrate Katta into Solr so that:
> * Distributed search uses Hadoop RPC
> * Shard/SolrCore distribution and management
> * Zookeeper based failover
> * Indexes may be built using Hadoop

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message