lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Collection API migrate statement
Date Tue, 15 Dec 2015 16:48:48 GMT
You might look at colleciton aliasing, this is sometimes used for
time-series data (which I'm guessing this is).

But I have to ask whether migrating tuf faround like that is really
necessary. 2M docs isn't very many, have you stress tested with just
indexing them all to a single collection? Is the traffic on the Hot
part really heavy enough to warrant the complexity?

Best,
Erick

On Tue, Dec 15, 2015 at 7:57 AM, philippa griggs
<philippa.griggs@hotmail.co.uk> wrote:
> Hello,
>
>
> Solr 5.2.1.
>
>
> I'm using the collection API migrate statement in our test environment with the view
to implement a Hot, Cold arrangement- newer documents will be kept on the Hot collection and
each night the oldest documents will be migrated into the Cold collection. I've got it all
working with a small amount of documents (around 28,000).
>
>
> I'm now trying to migrate around 200,000 documents and am getting 'migrate the collection
time out:180s'  message back.
>
>
> The logs from the source collection are:
>
>
> INFO  - 2015-12-15 14:43:19.183; [HotSessions   ] org.apache.solr.cloud.OverseerCollectionProcessor;
Successfully created replica of temp source collection on target leader node
> INFO  - 2015-12-15 14:43:19.183; [HotSessions   ] org.apache.solr.cloud.OverseerCollectionProcessor;
Requesting merge of temp source collection replica to target leader
> INFO  - 2015-12-15 14:45:36.648; [   ] org.apache.solr.cloud.DistributedQueue$LatchWatcher;
NodeDeleted fired on path /overseer/collection-queue-work/qnr-0000000004 state SyncConnected
> INFO  - 2015-12-15 14:45:36.651; [   ] org.apache.solr.cloud.DistributedQueue$LatchWatcher;
NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
> ERROR - 2015-12-15 14:45:36.651; [   ] org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
migrate the collection time out:180s
>         at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:237)
>         etc
>
>
> The logs from the target collection are:
>
> INFO  - 2015-12-15 14:43:19.128; [split_shard1_temp_shard2 shard1  split_shard1_temp_shard2_shard1_replica2]
org.apache.solr.update.UpdateLog; Took 22 ms to seed version buckets with highest version
1520634636692094979
> INFO  - 2015-12-15 14:43:19.129; [split_shard1_temp_shard2 shard1  split_shard1_temp_shard2_shard1_replica2]
org.apache.solr.cloud.RecoveryStrategy; Finished recovery process. core=split_shard1_temp_shard2_shard1_replica2
> INFO  - 2015-12-15 14:43:19.199; [   ] org.apache.solr.update.DirectUpdateHandler2; start
mergeIndexes{}
>
> As there are no errors in the target collection, am I right in assuming the timeout occured
because the merge took too long? If that is so, how to I increase the timeout period? Ideally
I will need to migrate around 2 million documents a night.
>
>
> Any help would be much appreciated.
>
>
> Philippa
>
>

Mime
View raw message