lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Per Steffensen <st...@designware.dk>
Subject Re: Exposing Solr routing to SolrJ client
Date Wed, 14 Mar 2012 12:43:52 GMT
FYI (if it is of any interest), we just hacked CloudSolrServer locally 
to support routing of realtime-get requests. Limitations are:
- Only "id"-parameter and not "ids"-parameter supported in realtime-get 
requests.
- Only schemas with uniqueKey on field named "id" and only "id"-field of 
type string supported.

We did this to be able to start performance tests on our own system 
building on SolrCloud. The performance of our own system is dependent on 
being able to do realtime-gets from the client (our system), because we 
often do updates of documents very quickly after they have been indexed 
for the first time (and we run with soft-commit = 1 sec - we cant wait 
for that). We use the "version control" (for optimistic locking) and 
"unique key constraint where you fail instead of overwrite if document 
already exists" 
(http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics) in our 
highly concurrent performance test, so that will also be tested wrt 
performance.

What we did in CloudSolrServer was:
* Added the following to the requst method between the "if (collection 
== null)" statement and the "LBHttpSolrServer.Req req = new 
LBHttpSolrServer.Req(request, urlList);" statement:
    List<String> urlList = new ArrayList<String>();
    if (reqParams.get(CommonParams.QT) != null && 
reqParams.get(CommonParams.QT).equals("/get")) {
      String id = reqParams.get("id");
      int hash = hash(id);
      String shardId = getShard(hash, collection, cloudState);
     
      ZkCoreNodeProps leaderProps = null;
      try {
        leaderProps = new ZkCoreNodeProps(zkStateReader.getLeaderProps(
                collection, shardId));
      } catch (InterruptedException ie) {
        throw new SolrServerException(ie);
      }
 
      String fullUrl = 
ensureUrlHasProtocolIdentifier(leaderProps.getCoreUrl());
     
      urlList.add(fullUrl);
    } else {
      <stuff that was already in request between the "if (collection == 
null)" statement and the "LBHttpSolrServer.Req req = new 
LBHttpSolrServer.Req(request, urlList);" statement
    }
* Added the follow helper-methods (stolen from 
DistributedUpdateProcessor etc.)
  private String ensureUrlHasProtocolIdentifier(String url) {
    if (!url.startsWith("http://") && !url.startsWith("https://")) {
      url = "http://" + url;
    }
    return url;
  }
 
  private String getShard(int hash, String collection, CloudState 
cloudState) {
    return cloudState.getShard(hash, collection);
  }
 
  private int hash(String id) {
    BytesRef indexedId = new BytesRef();
    UnicodeUtil.UTF16toUTF8(id, 0, id.length(), indexedId);
    return Hash.murmurhash3_x86_32(indexedId.bytes, indexedId.offset, 
indexedId.length, 0);
  }

It seems to work for us, but we look very much forward to the "real" 
solution.

Regards, Per Steffensen


Per Steffensen skrev:
>
>> Right, you can't yet even with CloudSolrServer - but I think it will 
>> be done soon - certainly before the 4 release anyway.
> Ok, I will cross my fingers for it to be done soon. Thanks for your 
> kind help.
>
> Regards, Steff
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message