lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hrishikesh Gadre <gadre.s...@gmail.com>
Subject Re: Challenges with new Solrcloud Backup/Restore functionality
Date Tue, 08 Nov 2016 21:15:10 GMT
Hi Stephen,

Thanks for the update.

Regarding SOLR-9527 - I think we need a unit test for verifying
"createNodeSet" functionality. I will spend some time on it in next couple
of days.

Also regarding #2, i also found similar issue (doc count mismatch after
restore) while testing with a large collection (~50GB index size). I have
opened SOLR-9598 to track this. Please take a look and comment if you have
any insight.

-Hrishikesh

On Tue, Nov 8, 2016 at 12:54 PM, Stephen Weiss <Steve.Weiss@wgsn.com> wrote:

> Just wanted to note that we tested out the patch from SOLR-9527 and it
> worked perfectly for the balancing issue - thank you so much for that!
>
> As for issue #2, we've resorted to doing a hard commit, stopping all
> indexing against the index, and then taking the backup, and we have a
> reasonably good success rate with that.  The system is set up to
> automatically delete and retry the backup/restore process if the cores
> don't match, so that's allowed us to smooth over that problem and get this
> process out into production.   We've been using it for several weeks now
> without any major issue!
>
> We just looked because Solr 6.3 was out, and wanted to know if we could
> upgrade without patching again, but it appears this ticket hasn't gone
> anywhere yet.  I know one users' testing is probably not enough, but given
> that it seems the patch works just fine, are there any plans to merge it
> into release yet?
>
> --
> Steve
>
> On Tue, Oct 4, 2016 at 6:46 PM, Stephen Lewis <slewis@panopto.com<mailto:
> slewis@panopto.com>> wrote:
> Hi All,
>
> I have been experiencing error#1 too with the current branch_6_2 build. I
> started noticing after I applied my patch to that branch<
> https://issues.apache.org/jira/browse/SOLR-9527> (on issue #2), but it
> appears to occur without the patch as well. I haven't seen this issue with
> solr 6.1.0 despite extensive testing. I haven't confirmed if this occurs on
> the official 6.2.0 release build. I will try to confirm and gather more
> data soon.
>
> As with Stephen Weiss, I also am not seeing any errors logged in the index
> after backup and the task is marked as succeeded. However, after each
> backup which is missing a large amount of data, the restore command fails,
> in the sense that the collection is created, but the initialized cores are
> blank and the logs contain errors about "incomplete segments". I will try
> to research further and get back with more data soon.
>
>
>
> On Mon, Sep 26, 2016 at 11:26 AM, Hrishikesh Gadre <gadre.solr@gmail.com
> <mailto:gadre.solr@gmail.com>> wrote:
> Hi Stephen,
>
> regarding #1, can you verify following steps during backup/restore?
>
> - Before backup command, make sure to run a "hard" commit on the original
> collection. The backup operation will capture only hard committed data.
> - After restore command, check the Solr web UI to verify that all replicas
> of the new (or restored) collection are in the "active" state. During my
> testing, I found that when one or more replicas are in "recovery" state,
> the doc count of the restored collection doesn't match the doc count of the
> original collection. But after the recovery is complete, the doc counts
> match. I will file a JIRA to fix this issue.
>
> Thanks
> Hrishikesh
>
> On Mon, Sep 26, 2016 at 9:34 AM, Stephen Weiss <Steve.Weiss@wgsn.com
> <mailto:Steve.Weiss@wgsn.com>> wrote:
>
> > #2 - that's great news.  I'll try to patch it in and test it out.
> >
> > #1 - In all cases, the backup and restore both appear successful.  There
> > are no failure messages for any of the shards, no warnings, etc - I
> didn't
> > even realize at first that data was missing until I noticed differences
> in
> > some of the query results when we were testing.  Either manual restore of
> > the data or using the restore API (with all data on one node), we see the
> > same, so I think it's more a problem in the backup process than the
> restore
> > process.
> >
> > If there's any kind of debugging output we can provide that can help
> solve
> > this, let me know.
> >
> > --
> > Steve
> >
> > On Sun, Sep 25, 2016 at 7:17 PM, Hrishikesh Gadre <gadre.solr@gmail.com
> <mailto:gadre.solr@gmail.com>>
> > wrote:
> >
> >> Hi Steve,
> >>
> >> Regarding the 2nd issue, a JIRA is already created and patch is uploaded
> >> (SOLR-9527). Can someone review and commit the patch?
> >>
> >> Regarding 1st issue, does backup command succeeds? Also do you see any
> >> warning/error log messages? How about the restore command?
> >>
> >> Thanks
> >> Hrishikesh
> >>
> >>
> >>
> >> On Sat, Sep 24, 2016 at 12:14 PM, Stephen Weiss <Steve.Weiss@wgsn.com
> <mailto:Steve.Weiss@wgsn.com>>
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> We're very excited about SolrCloud's new backup / restore collection
> >>> APIs, which should introduce some major new efficiencies into our
> indexing
> >>> workflow.  Unfortunately, we've run into some snags with it that are
> >>> preventing us from moving into production.  I was hoping someone on the
> >>> list could help.
> >>>
> >>> 1) Data inconsistencies
> >>>
> >>> There seems to be a problem getting all the data consistently.
> >>> Sometimes, the backup will contain all of the data in the collection,
> and
> >>> sometimes, large portions of the collection (as much as 40%) will be
> >>> missing.  We haven't quite figured out what might cause this yet,
> although
> >>> one thing I've noticed is the chances of success are greater when we
> are
> >>> only backing up one collection at a time.  Unfortunately, for our
> workflow,
> >>> it will be difficult to make that work, and there still doesn't seem
> to be
> >>> a guarantee of success either way.
> >>>
> >>> 2) Shards are not distributed
> >>>
> >>> To make matters worse, for some reason, any type of restore operation
> >>> always seems to put all shards of the collection on the same node.
> We've
> >>> tried setting maxShardsPerNode to 1 in the restore command, but this
> has no
> >>> effect.  We are seeing the same behavior on both 6.1 and 6.2.1.  No
> matter
> >>> what we do, all the shards always go to the same node - and it's not
> even
> >>> the node that we execute the restore request on, but oddly enough, a
> >>> totally different node, and always the same one (the 4th one).  It
> should
> >>> be noted that all nodes of our 8 node cloud are up and totally
> functional
> >>> when this happens.
> >>>
> >>> To work around this, we wrote up a quick script to create an empty
> >>> collection, which always distributes itself across the cloud quite well
> >>> (another indication that there's nothing wrong with the nodes
> themselves),
> >>> and then we rsync the individual shards' data into the empty shards and
> >>> reload the collection.  This works fine, however, because of the data
> >>> inconsistencies mentioned above, we can't really move forward anyway.
> >>>
> >>>
> >>> Problem #2, we have a reasonable workaround for, but problem #1 we do
> >>> not.  If anyone has any thoughts about either of these problems, I
> would be
> >>> very grateful.  Thanks!
> >>>
> >>> --
> >>> Steve
> >>>
> >>> ________________________________
> >>>
> >>> WGSN is a global foresight business. Our experts provide deep insight
> >>> and analysis of consumer, fashion and design trends. We inspire our
> clients
> >>> to plan and trade their range with unparalleled confidence and
> accuracy.
> >>> Together, we Create Tomorrow.
> >>>
> >>> WGSN<http://www.wgsn.com/> is part of WGSN Limited, comprising of
> >>> market-leading products including WGSN.com<http://www.wgsn.com>, WGSN
> >>> Lifestyle & Interiors<http://www.wgsn.com/en/lifestyle-interiors>,
> WGSN
> >>> INstock<http://www.wgsninstock.com/>, WGSN StyleTrial<
> >>> http://www.wgsn.com/en/styletrial/> and WGSN Mindset<
> >>> http://www.wgsn.com/en/services/consultancy/>, our bespoke consultancy
> >>> services.
> >>>
> >>> The information in or attached to this email is confidential and may be
> >>> legally privileged. If you are not the intended recipient of this
> message,
> >>> any use, disclosure, copying, distribution or any action taken in
> reliance
> >>> on it is prohibited and may be unlawful. If you have received this
> message
> >>> in error, please notify the sender immediately by return email and
> delete
> >>> this message and any copies from your computer and network. WGSN does
> not
> >>> warrant that this email and any attachments are free from viruses and
> >>> accepts no liability for any loss resulting from infected email
> >>> transmissions.
> >>>
> >>> WGSN reserves the right to monitor all email through its networks. Any
> >>> views expressed may be those of the originator and not necessarily of
> WGSN.
> >>> WGSN is powered by Ascential plc<http://www.ascential.com>, which
> >>> transforms knowledge businesses to deliver exceptional performance.
> >>>
> >>> Please be advised all phone calls may be recorded for training and
> >>> quality purposes and by accepting and/or making calls from and/or to
> us you
> >>> acknowledge and agree to calls being recorded.
> >>>
> >>> WGSN Limited, Company number 4858491
> >>>
> >>> registered address:
> >>>
> >>> Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP
> >>>
> >>> WGSN Inc., tax ID 04-3851246, registered office c/o National Registered
> >>> Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United
> States
> >>>
> >>> 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register):
> >>> 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP
> >>> 01453-000, Itaim Bibi, São Paulo
> >>>
> >>> 4C Business Information Consulting (Shanghai) Co., Ltd,
> >>> 富新商务信息咨询(上海)有限公司, registered address Unit
4810/4811, 48/F Tower 1,
> Grand
> >>> Gateway, 1 Hong Qiao Road, Xuhui District, Shanghai
> >>>
> >>
> >>
> >
> > ------------------------------
>
> >
> > WGSN is a global foresight business. Our experts provide deep insight and
> > analysis of consumer, fashion and design trends. We inspire our clients
> to
> > plan and trade their range with unparalleled confidence and accuracy.
> > Together, we Create Tomorrow.
> >
> > WGSN <http://www.wgsn.com/> is part of WGSN Limited, comprising of
> > market-leading products including WGSN.com <http://www.wgsn.com>, WGSN
> > Lifestyle & Interiors <http://www.wgsn.com/en/lifestyle-interiors>, WGSN
> > INstock <http://www.wgsninstock.com/>, WGSN StyleTrial
> > <http://www.wgsn.com/en/styletrial/> and WGSN Mindset
> > <http://www.wgsn.com/en/services/consultancy/>, our bespoke consultancy
> > services.
> >
> > The information in or attached to this email is confidential and may be
> > legally privileged. If you are not the intended recipient of this
> message,
> > any use, disclosure, copying, distribution or any action taken in
> reliance
> > on it is prohibited and may be unlawful. If you have received this
> message
> > in error, please notify the sender immediately by return email and delete
> > this message and any copies from your computer and network. WGSN does not
> > warrant that this email and any attachments are free from viruses and
> > accepts no liability for any loss resulting from infected email
> > transmissions.
> >
> > WGSN reserves the right to monitor all email through its networks. Any
> > views expressed may be those of the originator and not necessarily of
> WGSN.
> > WGSN is powered by Ascential plc <http://www.ascential.com>, which
> > transforms knowledge businesses to deliver exceptional performance.
> >
> > Please be advised all phone calls may be recorded for training and
> quality
> > purposes and by accepting and/or making calls from and/or to us you
> > acknowledge and agree to calls being recorded.
> >
> > WGSN Limited, Company number 4858491
> >
> > registered address:
> >
> > Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP
> >
> > WGSN Inc., tax ID 04-3851246, registered office c/o National Registered
> > Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United
> States
> >
> > 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register):
> > 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP
> > 01453-000, Itaim Bibi, São Paulo
> >
> > 4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询(上海)有限公司,
> > registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong
> Qiao
> > Road, Xuhui District, Shanghai
> >
>
>
>
> --
> Stephen
>
> (206)753-9320<tel:%28206%29753-9320>
> stephen-lewis.net<http://stephen-lewis.net>
>
>
> ________________________________
>
> WGSN is a global foresight business. Our experts provide deep insight and
> analysis of consumer, fashion and design trends. We inspire our clients to
> plan and trade their range with unparalleled confidence and accuracy.
> Together, we Create Tomorrow.
>
> WGSN<http://www.wgsn.com/> is part of WGSN Limited, comprising of
> market-leading products including WGSN.com<http://www.wgsn.com>, WGSN
> Lifestyle & Interiors<http://www.wgsn.com/en/lifestyle-interiors>, WGSN
> INstock<http://www.wgsninstock.com/>, WGSN StyleTrial<http://www.wgsn.
> com/en/styletrial/> and WGSN Mindset<http://www.wgsn.com/
> en/services/consultancy/>, our bespoke consultancy services.
>
> The information in or attached to this email is confidential and may be
> legally privileged. If you are not the intended recipient of this message,
> any use, disclosure, copying, distribution or any action taken in reliance
> on it is prohibited and may be unlawful. If you have received this message
> in error, please notify the sender immediately by return email and delete
> this message and any copies from your computer and network. WGSN does not
> warrant that this email and any attachments are free from viruses and
> accepts no liability for any loss resulting from infected email
> transmissions.
>
> WGSN reserves the right to monitor all email through its networks. Any
> views expressed may be those of the originator and not necessarily of WGSN.
> WGSN is powered by Ascential plc<http://www.ascential.com>, which
> transforms knowledge businesses to deliver exceptional performance.
>
> Please be advised all phone calls may be recorded for training and quality
> purposes and by accepting and/or making calls from and/or to us you
> acknowledge and agree to calls being recorded.
>
> WGSN Limited, Company number 4858491
>
> registered address:
>
> Ascential plc, The Prow, 1 Wilder Walk, London W1B 5AP
>
> WGSN Inc., tax ID 04-3851246, registered office c/o National Registered
> Agents, Inc., 160 Greentree Drive, Suite 101, Dover DE 19904, United States
>
> 4C Serviços de Informação Ltda., CNPJ/MF (Taxpayer's Register):
> 15.536.968/0001-04, Address: Avenida Cidade Jardim, 377, 7˚ andar CEP
> 01453-000, Itaim Bibi, São Paulo
>
> 4C Business Information Consulting (Shanghai) Co., Ltd, 富新商务信息咨询(上海)有限公司,
> registered address Unit 4810/4811, 48/F Tower 1, Grand Gateway, 1 Hong Qiao
> Road, Xuhui District, Shanghai
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message