lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Weiss <Steve.We...@wgsn.com>
Subject Challenges with new Solrcloud Backup/Restore functionality
Date Sat, 24 Sep 2016 19:14:10 GMT
Hi everyone,

We're very excited about SolrCloud's new backup / restore collection APIs, which should introduce
some major new efficiencies into our indexing workflow.  Unfortunately, we've run into some
snags with it that are preventing us from moving into production.  I was hoping someone on
the list could help.

1) Data inconsistencies

There seems to be a problem getting all the data consistently.  Sometimes, the backup will
contain all of the data in the collection, and sometimes, large portions of the collection
(as much as 40%) will be missing.  We haven't quite figured out what might cause this yet,
although one thing I've noticed is the chances of success are greater when we are only backing
up one collection at a time.  Unfortunately, for our workflow, it will be difficult to make
that work, and there still doesn't seem to be a guarantee of success either way.

2) Shards are not distributed

To make matters worse, for some reason, any type of restore operation always seems to put
all shards of the collection on the same node.  We've tried setting maxShardsPerNode to 1
in the restore command, but this has no effect.  We are seeing the same behavior on both 6.1
and 6.2.1.  No matter what we do, all the shards always go to the same node - and it's not
even the node that we execute the restore request on, but oddly enough, a totally different
node, and always the same one (the 4th one).  It should be noted that all nodes of our 8 node
cloud are up and totally functional when this happens.

To work around this, we wrote up a quick script to create an empty collection, which always
distributes itself across the cloud quite well (another indication that there's nothing wrong
with the nodes themselves), and then we rsync the individual shards' data into the empty shards
and reload the collection.  This works fine, however, because of the data inconsistencies
mentioned above, we can't really move forward anyway.


Problem #2, we have a reasonable workaround for, but problem #1 we do not.  If anyone has
any thoughts about either of these problems, I would be very grateful.  Thanks!

--
Steve

________________________________

WGSN is a global foresight business. Our experts provide deep insight and analysis of consumer,
fashion and design trends. We inspire our clients to plan and trade their range with unparalleled
confidence and accuracy. Together, we Create Tomorrow.

WGSN<http://www.wgsn.com/> is part of WGSN Limited, comprising of market-leading products
including WGSN.com<http://www.wgsn.com>, WGSN Lifestyle & Interiors<http://www.wgsn.com/en/lifestyle-interiors>,
WGSN INstock<http://www.wgsninstock.com/>, WGSN StyleTrial<http://www.wgsn.com/en/styletrial/>
and WGSN Mindset<http://www.wgsn.com/en/services/consultancy/>, our bespoke consultancy
services.

The information in or attached to this email is confidential and may be legally privileged.
If you are not the intended recipient of this message, any use, disclosure, copying, distribution
or any action taken in reliance on it is prohibited and may be unlawful. If you have received
this message in error, please notify the sender immediately by return email and delete this
message and any copies from your computer and network. WGSN does not warrant that this email
and any attachments are free from viruses and accepts no liability for any loss resulting
from infected email transmissions.

WGSN reserves the right to monitor all email through its networks. Any views expressed may
be those of the originator and not necessarily of WGSN. WGSN is powered by Ascential plc<http://www.ascential.com>,
which transforms knowledge businesses to deliver exceptional performance.

Please be advised all phone calls may be recorded for training and quality purposes and by
accepting and/or making calls from and/or to us you acknowledge and agree to calls being recorded.

WGSN Limited, Company number 4858491

registered address:

Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP

WGSN Inc., tax ID 04-3851246, registered office c/o National Registered Agents, Inc., 160
Greentree Drive, Suite 101, Dover DE 19904, United States

4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register): 15.536.968/0001-04, Address:
Avenida Cidade Jardim, 377, 7˚ andar CEP 01453-000, Itaim Bibi, São Paulo

4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询(上海)有限公司,
registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong Qiao Road, Xuhui District,
Shanghai

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message