lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-4998) Make the use of Slice and Shard consistent across the code and document base
Date Tue, 09 Jul 2013 22:43:50 GMT

    [ https://issues.apache.org/jira/browse/SOLR-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13703917#comment-13703917
] 

Jack Krupansky commented on SOLR-4998:
--------------------------------------

bq. The "maxShardsPerNode" parameter uses "shard" in the incorrect sense I believe?

Uhhh... and I was under the impression that it was using it the "right" way! Sigh.

I wasn't following SOLR-4114 closely enough, but I do recall that when I read "it is much
easier to move an entire existing shards from one Solr server to another one that just joined
the cluter than it is to split an exsiting shard among the Solr that used to run it and the
new Solr", I [mis]interpreted that as meaning that shard meant slice or logical shard and
that I could have multiple logical shard/slice key ranges on a single node, so that as shards/slices
got too big for that single node, they could be trivially moved to new nodes. In other words,
that each node could support multiple key hash ranges. For example, start with 64 logical
shards on 4 machines at 16 logical shards per node, and then be able to incrementally grow
up to 64 logical shards on 64 nodes without any shard splitting needed. Boy, was I wrong!
Wow.

Honestly, from my recollections of the email discussions at the time, I would have sworn on
a stack of bibles that Per, et al were talking about (logical) shards, NOT what we now refer
to as replicas or physical shards.

See:
https://issues.apache.org/jira/browse/SOLR-4114
SOLR-4114 - Collection API: Allow multiple shards from one collection on the same Solr server

Shame on me for not reading the patch carefully to see what I see now in the code:

{code}
      int maxShardsAllowedToCreate = maxShardsPerNode * nodeList.size();
      int requestedShardsToCreate = numSlices * repFactor;
{code}

When what I expected was something like:

{code}
      int requestedLogicalShardsPerNode = (numSlices + maxShardsPerNode - 1) / maxShardsPerNode;
{code}

I mean, it was already easy enough to add replica nodes for SolrCloud anyway, so the ultimate
implementation did not add a great value to SolrCloud, compared to the value it would have
added if multiple logical shards could have been supported per node.

OTOH, maybe the overhead for "SplitShard" is modest enough that it delivers the same final
value - scaling document capacity, as opposed to scaling query capacity which SOLR 4114 now
seems more focused on.

Oh well...

                
> Make the use of Slice and Shard consistent across the code and document base
> ----------------------------------------------------------------------------
>
>                 Key: SOLR-4998
>                 URL: https://issues.apache.org/jira/browse/SOLR-4998
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>    Affects Versions: 4.3, 4.3.1
>            Reporter: Anshum Gupta
>
> The interchangeable use of Slice and Shard is pretty confusing at times. We should define
each separately and use the apt term whenever we do so.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message