lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: optimize status
Date Tue, 30 Jun 2015 15:42:37 GMT
On 6/29/2015 2:48 PM, Reitzel, Charles wrote:
> I take your point about shards and segments being different things.  I understand that
the hash ranges per segment are not kept in ZK.   I guess I wish they were.
>
> In this regard, I liked Mongodb, uses a 2-level sharding scheme.   Each shard manages
a list of  "chunks", each has its own hash range which is kept in the cluster state.   If
data needs to be balanced across nodes, it works at the chunk level.  No record/doc level
I/O is necessary.   Much more targeted and only the data that needs to move is touched.  Solr
does most things better than Mongo, imo.  But this is one area where the Mongo got it right.

Segment detail would not only lead to a data explosion in the
clusterstate, it would be crossing abstraction boundaries, and would
potentially require updating the clusterstate just because a single
document was inserted into the index.  That one tiny update could (and
probably would) create a new segment on one shard.  Due to the way
SolrCloud replicates data during normal operation, every replica for a
given shard might have a different set of segments, which means segments
would need to be tracked at the replica level, not the shard level.

Also, Solr cannot control which hash ranges end up in each segment. 
Solr only knows about the index as a whole ... implementation details
like segments are left entirely up to Lucene, and although I admit to
not knowing Lucene internals very well, I don't think Lucene offers any
way to control that either.  You mention that MongoDB dictates which
hash ranges end up in each chunk.  That implies that MongoDB can control
each chunk.  If we move the analogy to Solr, it breaks down because Solr
cannot control segments.  Although Solr does have several configuration
knobs that affect how segments are created, those configurations are
simply passed through to Lucene, Solr itself does not use that information.

Thanks,
Shawn


Mime
View raw message