Return-Path: X-Original-To: apmail-lucene-solr-user-archive@minotaur.apache.org Delivered-To: apmail-lucene-solr-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A508EE349 for ; Wed, 19 Dec 2012 12:57:46 +0000 (UTC) Received: (qmail 57681 invoked by uid 500); 19 Dec 2012 12:57:43 -0000 Delivered-To: apmail-lucene-solr-user-archive@lucene.apache.org Received: (qmail 57626 invoked by uid 500); 19 Dec 2012 12:57:42 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 57617 invoked by uid 99); 19 Dec 2012 12:57:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2012 12:57:42 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of erickerickson@gmail.com designates 209.85.214.179 as permitted sender) Received: from [209.85.214.179] (HELO mail-ob0-f179.google.com) (209.85.214.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2012 12:57:33 +0000 Received: by mail-ob0-f179.google.com with SMTP id x4so1903059obh.24 for ; Wed, 19 Dec 2012 04:57:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=98rH1iFeDn9UCZrKGekxDstgFCnNv4xxpSkO7xPwuB8=; b=LV7+fdGPt/D/fHUKFy9v2OElNwhs6zAiGTHgAWxUOP7FQGktleCilit//pFac1giwB nBZUTM3DjUlaxlVbe9nimyY+NYooFeb1x907XC+KV350gBRB/J8rhsiFBBO+dcwxQy2G M2rSAXrCvfqcuRvwS7ca8xxr2gL+TvLiXBVID1ErnwLrY2QeSvgAeZjn4VeVUnzJw/RK SubEZThWJb0/ds6QpScg2aybwZ6UUFPLzJVeRF83Pg4KLGT363BlX4PxdPv0ySqbYwA+ ikswk8x9fxEJ1074q2bR/gzeh2eSyiXXyN+lOiJFIGFg/lD5zldjzpDpOfbNcmPefXkZ cPWA== MIME-Version: 1.0 Received: by 10.60.169.76 with SMTP id ac12mr4589125oec.137.1355921832334; Wed, 19 Dec 2012 04:57:12 -0800 (PST) Received: by 10.60.32.78 with HTTP; Wed, 19 Dec 2012 04:57:12 -0800 (PST) In-Reply-To: References: <0CBF0EF7-30ED-4388-B5BF-62A1DE5811E9@gmail.com> Date: Wed, 19 Dec 2012 07:57:12 -0500 Message-ID: Subject: Re: Strange data-loss problem on one of our cores From: Erick Erickson To: solr-user@lucene.apache.org Content-Type: multipart/alternative; boundary=bcaec517a57228d62404d1342706 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec517a57228d62404d1342706 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks for letting us know, and do bring let us know if you see the problem again. Erick On Tue, Dec 18, 2012 at 7:39 AM, John Nielsen wrote: > I build a solr version from the solr-4x branch yesterday and so far am > unable to replicate the problems i had before. > > I am cautiously optimistic that the problem has been resolved. If i run > into any more problems, I'll let you all know. > > > -- > Med venlig hilsen / Best regards > > *John Nielsen* > Programmer > > > > *MCB A/S* > Enghaven 15 > DK-7500 Holstebro > > Kundeservice: +45 9610 2824 > post@mcb.dk > www.mcb.dk > > > > On Fri, Dec 14, 2012 at 7:33 PM, Markus Jelsma > wrote: > > > Mark, no issue has been filed. That cluster runs a check out from round > > end of july/beginning of august. I'm in the process of including anothe= r > > cluster in the indexing and removal of documents besides the old > production > > clusters. I'll start writing to that one tuesday orso. > > If i notice a discrepancy after some time i am sure to report it. I dou= bt > > i'll find it before 2013, if the problem is still there. > > > > > > -----Original message----- > > > From:Mark Miller > > > Sent: Fri 14-Dec-2012 19:05 > > > To: solr-user@lucene.apache.org > > > Subject: Re: Strange data-loss problem on one of our cores > > > > > > Have you filed a JIRA issue for this that I don't remember Markus? > > > > > > We need to make sure this is fixed. > > > > > > Any idea around when the trunk version came from? Before or after 4.0= ? > > > > > > - Mark > > > > > > On Dec 14, 2012, at 6:36 AM, Markus Jelsma > > > wrote: > > > > > > > We did not solve it but reindexing can remedy the problem. > > > > > > > > -----Original message----- > > > >> From:John Nielsen > > > >> Sent: Fri 14-Dec-2012 12:31 > > > >> To: solr-user@lucene.apache.org > > > >> Subject: Re: Strange data-loss problem on one of our cores > > > >> > > > >> How did you solve the problem? > > > >> > > > >> > > > >> -- > > > >> Med venlig hilsen / Best regards > > > >> > > > >> *John Nielsen* > > > >> Programmer > > > >> > > > >> > > > >> > > > >> *MCB A/S* > > > >> Enghaven 15 > > > >> DK-7500 Holstebro > > > >> > > > >> Kundeservice: +45 9610 2824 > > > >> post@mcb.dk > > > >> www.mcb.dk > > > >> > > > >> > > > >> > > > >> On Fri, Dec 14, 2012 at 12:04 PM, Markus Jelsma > > > >> wrote: > > > >> > > > >>> FYI, we observe the same issue, after some time (days, months) a > > cluster > > > >>> running an older trunk version has at least two shards where the > > leader and > > > >>> the replica do not contain the same number of records. No recover= y > is > > > >>> attempted, it seems it thinks everything is alright. Also, one co= re > > of one > > > >>> of the unsynced shards waits forever loading > > > >>> /replication?command=3Ddetail&wt=3Djson, other cores load it in a= few > > ms. Both > > > >>> cores of another unsynced shard does not show this problem. > > > >>> > > > >>> -----Original message----- > > > >>>> From:John Nielsen > > > >>>> Sent: Fri 14-Dec-2012 11:50 > > > >>>> To: solr-user@lucene.apache.org > > > >>>> Subject: Re: Strange data-loss problem on one of our cores > > > >>>> > > > >>>> I did a manual commit, and we are still missing docs, so it > doesn't > > look > > > >>>> like the search race condition you mention. > > > >>>> > > > >>>> My boss wasn't happy when i mentioned that I wanted to try out > > unreleased > > > >>>> code. Ill get him won over though and return with my findings. I= t > > will > > > >>>> probably be some time next week. > > > >>>> > > > >>>> Thanks for your help. > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Med venlig hilsen / Best regards > > > >>>> > > > >>>> *John Nielsen* > > > >>>> Programmer > > > >>>> > > > >>>> > > > >>>> > > > >>>> *MCB A/S* > > > >>>> Enghaven 15 > > > >>>> DK-7500 Holstebro > > > >>>> > > > >>>> Kundeservice: +45 9610 2824 > > > >>>> post@mcb.dk > > > >>>> www.mcb.dk > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Thu, Dec 13, 2012 at 4:10 PM, Mark Miller < > markrmiller@gmail.com > > > > > > >>> wrote: > > > >>>> > > > >>>>> Couple things to start: > > > >>>>> > > > >>>>> By default SolrCloud distributes updates a doc at a time. So if > you > > > >>> have 1 > > > >>>>> shard, whatever node you index too, it will send updates to the > > other. > > > >>>>> Replication is only used for recovery, not distributing data. S= o > > for > > > >>> some > > > >>>>> reason, there is an IOException when it tries to forward. > > > >>>>> > > > >>>>> The other issue is not something that Ive seen reported. Can/di= d > > you > > > >>> try > > > >>>>> and do another hard commit to make sure you had the latest sear= ch > > open > > > >>> when > > > >>>>> checking the # of docs on each node? There was previously a rac= e > > around > > > >>>>> commit that could cause some issues around expected visibility. > > > >>>>> > > > >>>>> If you are able to, you might try out a nightly build - 4.1 wil= l > be > > > >>> ready > > > >>>>> very soon and has numerous bug fixes for SolrCloud. > > > >>>>> > > > >>>>> - Mark > > > >>>>> > > > >>>>> On Dec 13, 2012, at 9:53 AM, John Nielsen wrote: > > > >>>>> > > > >>>>>> Hi all, > > > >>>>>> > > > >>>>>> We are seeing a strange problem on our 2-node solr4 cluster. > This > > > >>> problem > > > >>>>>> has resultet in data loss. > > > >>>>>> > > > >>>>>> We have two servers, varnish01 and varnish02. Zookeeper is > > running on > > > >>>>>> varnish02, but in a separate jvm. > > > >>>>>> > > > >>>>>> We index directly to varnish02 and we read from varnish01. Dat= a > is > > > >>> thus > > > >>>>>> replicated from varnish02 to varnish01. > > > >>>>>> > > > >>>>>> I found this in the varnish01 log: > > > >>>>>> > > > >>>>>> *INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish02.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DTOLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D42 > > > >>>>>> Dec 13, 2012 12:23:36 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish02.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DTOLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D41 > > > >>>>>> Dec 13, 2012 12:23:36 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish02.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DTOLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D33 > > > >>>>>> Dec 13, 2012 12:23:36 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish02.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DTOLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D33 > > > >>>>>> Dec 13, 2012 12:23:39 PM org.apache.solr.common.SolrException > log > > > >>>>>> SEVERE: shard update error StdNode: > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish02.lynero.net:8000/solr/default1_Norwegian/:org.apache.solr= .client.solrj.SolrServerException > > > >>>>> : > > > >>>>>> IOException occured when talking to server at: > > > >>>>>> http://varnish02.lynero.net:8000/solr/default1_Norwegian > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.j= ava:413) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.j= ava:181) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:= 335) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:= 309) > > > >>>>>> at > > > >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334= ) > > > >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > > >>>>>> at > > > >>>>>> > > > >>> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > >>>>>> at > > > >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334= ) > > > >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > > > >>>>>> at java.lang.Thread.run(Thread.java:636) > > > >>>>>> Caused by: org.apache.http.NoHttpResponseException: The target > > server > > > >>>>>> failed to respond > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponse= Parser.java:101) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser= .java:252) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(A= bstractHttpClientConnection.java:282) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(D= efaultClientConnection.java:247) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader= (AbstractClientConnAdapter.java:216) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpReques= tExecutor.java:298) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.= java:125) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequ= estDirector.java:647) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequest= Director.java:464) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient= .java:820) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient= .java:754) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient= .java:732) > > > >>>>>> at > > > >>>>>> > > > >>>>> > > > >>> > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.j= ava:352) > > > >>>>>> ... 11 more > > > >>>>>> > > > >>>>>> Dec 13, 2012 12:23:39 PM > > > >>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor > > doFinish > > > >>>>>> INFO: try and ask http://varnish02.lynero.net:8000/solr to > > recover* > > > >>>>>> > > > >>>>>> It looks like it is sending updates from varnish01 to > varnish02. I > > > >>> am not > > > >>>>>> sure for what since we only index on varnish02. Updates should > > never > > > >>> be > > > >>>>>> going from varnish01 to varnish02. > > > >>>>>> > > > >>>>>> Meanwhile on varnish02: > > > >>>>>> > > > >>>>>> *INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D16 > > > >>>>>> Dec 13, 2012 12:23:36 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D15 > > > >>>>>> Dec 13, 2012 12:23:36 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D16 > > > >>>>>> Dec 13, 2012 12:23:42 PM > > > >>> org.apache.solr.handler.admin.CoreAdminHandler > > > >>>>>> handleRequestRecoveryAction > > > >>>>>> INFO: It has been requested that we recover* > > > >>>>>> *Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execut= e > > > >>>>>> INFO: [default1_Danish] webapp=3D/solr path=3D/select > > > >>>>>> > > > >>>>> > > > >>> > > > params=3D{facet=3Dfalse&sort=3Ditem_group_59700_name_int+asc,+variant_of_= item_guid+asc&group.distributed.first=3Dtrue&facet.limit=3D1000&q.alt=3D*:*= &q.alt=3D*:*&distrib=3Dfalse&facet.method=3Denum&version=3D2&df=3Dtext&fl= =3Ddocid&shard.url=3D > > > >>>>>> > > > >>>>> > > > >>> > > > varnish02.lynero.net:8000/solr/default1_Danish/|varnish01.lynero.net:8000= /solr/default1_Danish/&NOW=3D1355397822111&group.field=3Dgroupby_variant_of= _item_guid&fq=3Dsite_guid:(11440)&fq=3Ditem_type:(PRODUCT)&fq=3Dlanguage_gu= id:(1)&fq=3Ditem_group_59700_combination:(*)&fq=3Ditem_group_45879_combinat= ion:(*)&fq=3Dis_searchable:(True)&querytype=3DTechnical&mm=3D100%25&facet.m= issing=3Don&group.ngroups=3Dtrue&facet.mincount=3D1&qf=3D%0a++++++++++text > > > >>>>> > > > >>> > > > ^0.5+name^1.2+searchable_text^0.8+typeahead_text^1.0+keywords^1.1+item_no= ^5.0%0a++++++++++ranking1_text^1.0+ranking2_text^2.0+ranking3_text^3.0%0a++= +++++&wt=3Djavabin&group.facet=3Dtrue&defType=3Dedismax&rows=3D0&facet.sort= =3Dlex&start=3D0&group=3Dtrue&group.sort=3Dname+asc&isShard=3Dtrue} > > > >>>>>> status=3D0 QTime=3D1 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Danish] webapp=3D/solr path=3D/select/ > > > >>>>>> params=3D{fq=3Dsite_guid:(2810678)&q=3Dwin} hits=3D0 status=3D= 0 QTime=3D17 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Danish] webapp=3D/solr path=3D/select > > > >>>>>> > > > >>>>> > > > >>> > > > params=3D{facet=3Don&sort=3Ditem_group_59700_name_int+asc,+variant_of_ite= m_guid+asc&q.alt=3D*:*&q.alt=3D*:*&distrib=3Dfalse&facet.method=3Denum&grou= p.distributed.second=3Dtrue&version=3D2&df=3Dtext&fl=3Ddocid&shard.url=3D > > > >>>>>> > > > >>>>> > > > >>> > > > varnish02.lynero.net:8000/solr/default1_Danish/|varnish01.lynero.net:8000= /solr/default1_Danish/&NOW=3D1355397822111&group.field=3Dgroupby_variant_of= _item_guid&fq=3Dsite_guid:(11440)&fq=3Ditem_type:(PRODUCT)&fq=3Dlanguage_gu= id:(1)&fq=3Ditem_group_59700_combination:(*)&fq=3Ditem_group_45879_combinat= ion:(*)&fq=3Dis_searchable:(True)&querytype=3DTechnical&mm=3D100%25&facet.m= issing=3Don&group.ngroups=3Dtrue&qf=3D%0a++++++++++text > > > >>>>> > > > >>> > > > ^0.5+name^1.2+searchable_text^0.8+typeahead_text^1.0+keywords^1.1+item_no= ^5.0%0a++++++++++ranking1_text^1.0+ranking2_text^2.0+ranking3_text^3.0%0a++= +++++&wt=3Djavabin&group.facet=3Dtrue&defType=3Dedismax&rows=3D0&facet.sort= =3Dlex&start=3D0&group=3Dtrue&group.sort=3Dname+asc&isShard=3Dtrue} > > > >>>>>> status=3D0 QTime=3D1 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Danish] webapp=3D/solr path=3D/select > > > >>>>>> > > > >>>>> > > > >>> > > > params=3D{facet=3Dfalse&sort=3Ditem_group_59700_name_int+asc,+variant_of_= item_guid+asc&group.distributed.first=3Dtrue&facet.limit=3D1000&q.alt=3D*:*= &q.alt=3D*:*&distrib=3Dfalse&facet.method=3Denum&version=3D2&df=3Dtext&fl= =3Ddocid&shard.url=3D > > > >>>>>> > > > >>>>> > > > >>> > > > varnish02.lynero.net:8000/solr/default1_Danish/|varnish01.lynero.net:8000= /solr/default1_Danish/&NOW=3D1355397822138&group.field=3Dgroupby_variant_of= _item_guid&fq=3Dsite_guid:(11440)&fq=3Ditem_type:(PRODUCT)&fq=3Dlanguage_gu= id:(1)&fq=3Ditem_group_59700_combination:(*)&fq=3Ditem_group_45879_combinat= ion:(*)&fq=3Dis_searchable:(True)&querytype=3DTechnical&mm=3D100%25&facet.m= issing=3Don&group.ngroups=3Dtrue&facet.mincount=3D1&qf=3D%0a++++++++++text > > > >>>>> > > > >>> > > > ^0.5+name^1.2+searchable_text^0.8+typeahead_text^1.0+keywords^1.1+item_no= ^5.0%0a++++++++++ranking1_text^1.0+ranking2_text^2.0+ranking3_text^3.0%0a++= +++++&wt=3Djavabin&group.facet=3Dtrue&defType=3Dedismax&rows=3D40&facet.sor= t=3Dlex&start=3D0&group=3Dtrue&group.sort=3Dname+asc&isShard=3Dtrue} > > > >>>>>> status=3D0 QTime=3D1 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Danish] webapp=3D/solr path=3D/select > > > >>>>>> > > > >>>>> > > > >>> > > > params=3D{facet=3Don&sort=3Ditem_group_59700_name_int+asc,+variant_of_ite= m_guid+asc&q.alt=3D*:*&q.alt=3D*:*&distrib=3Dfalse&facet.method=3Denum&grou= p.distributed.second=3Dtrue&version=3D2&df=3Dtext&fl=3Ddocid&shard.url=3D > > > >>>>>> > > > >>>>> > > > >>> > > > varnish02.lynero.net:8000/solr/default1_Danish/|varnish01.lynero.net:8000= /solr/default1_Danish/&NOW=3D1355397822138&group.field=3Dgroupby_variant_of= _item_guid&fq=3Dsite_guid:(11440)&fq=3Ditem_type:(PRODUCT)&fq=3Dlanguage_gu= id:(1)&fq=3Ditem_group_59700_combination:(*)&fq=3Ditem_group_45879_combinat= ion:(*)&fq=3Dis_searchable:(True)&querytype=3DTechnical&mm=3D100%25&facet.m= issing=3Don&group.ngroups=3Dtrue&group.topgroups.groupby_variant_of_item_gu= id=3D2963217&group.topgroups.groupby_variant_of_item_guid=3D2963223&group.t= opgroups.groupby_variant_of_item_guid=3D2963219&group.topgroups.groupby_var= iant_of_item_guid=3D2963220&group.topgroups.groupby_variant_of_item_guid=3D= 2963221&group.topgroups.groupby_variant_of_item_guid=3D2963222&group.topgro= ups.groupby_variant_of_item_guid=3D2963224&group.topgroups.groupby_variant_= of_item_guid=3D2963218&qf=3D%0a++++++++++text > > > >>>>> > > > >>> > > > ^0.5+name^1.2+searchable_text^0.8+typeahead_text^1.0+keywords^1.1+item_no= ^5.0%0a++++++++++ranking1_text^1.0+ranking2_text^2.0+ranking3_text^3.0%0a++= +++++&wt=3Djavabin&group.facet=3Dtrue&defType=3Dedismax&rows=3D40&facet.sor= t=3Dlex&start=3D0&group=3Dtrue&group.sort=3Dname+asc&isShard=3Dtrue} > > > >>>>>> status=3D0 QTime=3D1 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D26 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D22 > > > >>>>>> Dec 13, 2012 12:23:42 PM > > org.apache.solr.update.DefaultSolrCoreState > > > >>>>>> doRecovery > > > >>>>>> Dec 13, 2012 12:23:42 PM > > org.apache.solr.update.DefaultSolrCoreState > > > >>>>>> doRecovery > > > >>>>>> INFO: Running recovery - first canceling any ongoing recovery > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D25 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D24 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D20 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D25 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D23 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D21 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D23 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Norwegian] webapp=3D/solr path=3D/update > > > >>>>> params=3D{distrib.from=3D > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/&update.distrib= =3DFROMLEADER&wt=3Djavabin&version=3D2 > > > >>>>> } > > > >>>>>> status=3D0 QTime=3D16 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.cloud.RecoveryStrateg= y > > run > > > >>>>>> INFO: Starting recovery process. core=3Ddefault1_Norwegian > > > >>>>>> recoveringAfterStartup=3Dfalse > > > >>>>>> Dec 13, 2012 12:23:42 PM > > org.apache.solr.common.cloud.ZkStateReader > > > >>>>>> updateClusterState > > > >>>>>> INFO: Updating cloud state from ZooKeeper... > > > >>>>>> Dec 13, 2012 12:23:42 PM > > > >>>>>> org.apache.solr.update.processor.LogUpdateProcessor finish* > > > >>>>>> > > > >>>>>> And less than a second later: > > > >>>>>> > > > >>>>>> *Dec 13, 2012 12:23:42 PM org.apache.solr.cloud.RecoveryStrate= gy > > > >>>>> doRecovery > > > >>>>>> INFO: Attempting to PeerSync from > > > >>>>>> > > > >>>>> > > > >>> > > > http://varnish01.lynero.net:8000/solr/default1_Norwegian/core=3Ddefault1_= Norwegian > > > >>>>>> - recoveringAfterStartup=3Dfalse > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.update.PeerSync sync > > > >>>>>> INFO: PeerSync: core=3Ddefault1_Norwegian url=3D > > > >>>>>> http://varnish02.lynero.net:8000/solr START replicas=3D[ > > > >>>>>> http://varnish01.lynero.net:8000/solr/default1_Norwegian/] > > > >>> nUpdates=3D100 > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.update.PeerSync sync > > > >>>>>> WARNING: PeerSync: core=3Ddefault1_Norwegian url=3D > > > >>>>>> http://varnish02.lynero.net:8000/solr too many updates receive= d > > > >>> since > > > >>>>> start > > > >>>>>> - startingUpdates no longer overlaps with our currentUpdates > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.cloud.RecoveryStrateg= y > > > >>>>> doRecovery > > > >>>>>> INFO: PeerSync Recovery was not successful - trying replicatio= n. > > > >>>>>> core=3Ddefault1_Norwegian > > > >>>>>> Dec 13, 2012 12:23:42 PM org.apache.solr.cloud.RecoveryStrateg= y > > > >>>>> doRecovery > > > >>>>>> INFO: Starting Replication Recovery. core=3Ddefault1_Norwegian > > > >>>>>> Dec 13, 2012 12:23:42 PM > > > >>> org.apache.solr.client.solrj.impl.HttpClientUtil > > > >>>>>> createClient > > > >>>>>> INFO: Creating new http client, > > > >>>>>> > > > >>> > > config:maxConnections=3D128&maxConnectionsPerHost=3D32&followRedirects= =3Dfalse > > > >>>>>> Dec 13, 2012 12:23:42 PM > > org.apache.solr.common.cloud.ZkStateReader$2 > > > >>>>>> process > > > >>>>>> INFO: A cluster state change has occurred - updating...* > > > >>>>>> > > > >>>>>> State change on varnish01 at the same time: > > > >>>>>> > > > >>>>>> *Dec 13, 2012 12:23:42 PM > > > >>> org.apache.solr.common.cloud.ZkStateReader$2 > > > >>>>>> process > > > >>>>>> INFO: A cluster state change has occurred - updating...* > > > >>>>>> * > > > >>>>>> *And a few seconds later on varnish02, the recovery finishes: > > > >>>>>> * > > > >>>>>> Dec 13, 2012 12:23:48 PM org.apache.solr.cloud.RecoveryStrateg= y > > > >>>>> doRecovery > > > >>>>>> INFO: Replication Recovery was successful - registering as > Active. > > > >>>>>> core=3Ddefault1_Norwegian > > > >>>>>> Dec 13, 2012 12:23:48 PM org.apache.solr.cloud.RecoveryStrateg= y > > > >>>>> doRecovery > > > >>>>>> INFO: Finished recovery process. core=3Ddefault1_Norwegian > > > >>>>>> Dec 13, 2012 12:23:48 PM org.apache.solr.core.SolrCore execute > > > >>>>>> INFO: [default1_Danish] webapp=3D/solr path=3D/select > > > >>>>>> > > > >>>>> > > > >>> > > > params=3D{facet=3Dfalse&sort=3Ditem_group_56823_name_int+asc,+variant_of_= item_guid+asc&group.distributed.first=3Dtrue&facet.limit=3D1000&q.alt=3D*:*= &q.alt=3D*:*&distrib=3Dfalse&facet.method=3Denum&version=3D2&df=3Dtext&fl= =3Ddocid&shard.url=3D > > > >>>>>> > > > >>>>> > > > >>> > > > varnish02.lynero.net:8000/solr/default1_Danish/|varnish01.lynero.net:8000= /solr/default1_Danish/&NOW=3D1355397828395&group.field=3Dgroupby_variant_of= _item_guid&facet.field=3Ditemgroups_int_mv&fq=3Dsite_guid:(11440)&fq=3Ditem= _type:(PRODUCT)&fq=3Dlanguage_guid:(1)&fq=3Ditem_group_56823_combination:(*= )&fq=3Ditem_group_45879_combination:(*)&fq=3Dis_searchable:(True)&querytype= =3DTechnical&mm=3D100%25&facet.missing=3Don&group.ngroups=3Dtrue&facet.minc= ount=3D1&qf=3D%0a++++++++++text > > > >>>>> > > > >>> > > > ^0.5+name^1.2+searchable_text^0.8+typeahead_text^1.0+keywords^1.1+item_no= ^5.0%0a++++++++++ranking1_text^1.0+ranking2_text^2.0+ranking3_text^3.0%0a++= +++++&wt=3Djavabin&group.facet=3Dtrue&defType=3Dedismax&rows=3D0&facet.sort= =3Dlex&start=3D0&group=3Dtrue&group.sort=3Dname+asc&isShard=3Dtrue} > > > >>>>>> status=3D0 QTime=3D8 > > > >>>>>> Dec 13, 2012 12:23:48 PM > > org.apache.solr.common.cloud.ZkStateReader > > > >>>>>> updateClusterState > > > >>>>>> INFO: Updating cloud state from ZooKeeper... * > > > >>>>>> > > > >>>>>> Which is picked up on varnish01: > > > >>>>>> > > > >>>>>> *Dec 13, 2012 12:23:48 PM > > > >>> org.apache.solr.common.cloud.ZkStateReader$2 > > > >>>>>> process > > > >>>>>> INFO: A cluster state change has occurred - updating...* > > > >>>>>> > > > >>>>>> It looks like it replicated successfully, only it didnt. The > > > >>>>>> default1_Norwegian core on varnish01 now has 55.071 docs and t= he > > same > > > >>>>> core > > > >>>>>> on varnish02 has 35.088 docs. > > > >>>>>> > > > >>>>>> I checked the log files for both JVM's and no stop-the-world G= C > > were > > > >>>>> taking > > > >>>>>> place. > > > >>>>>> > > > >>>>>> There is also nothing in the zookeeper log of interest that I > can > > > >>> see. > > > >>>>>> > > > >>>>>> > > > >>>>>> -- > > > >>>>>> Med venlig hilsen / Best regards > > > >>>>>> > > > >>>>>> *John Nielsen* > > > >>>>>> Programmer > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> *MCB A/S* > > > >>>>>> Enghaven 15 > > > >>>>>> DK-7500 Holstebro > > > >>>>>> > > > >>>>>> Kundeservice: +45 9610 2824 > > > >>>>>> post@mcb.dk > > > >>>>>> www.mcb.dk > > > >>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > > --bcaec517a57228d62404d1342706--