Return-Path: X-Original-To: apmail-zookeeper-dev-archive@www.apache.org Delivered-To: apmail-zookeeper-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3C0F2717C for ; Fri, 5 Aug 2011 16:34:30 +0000 (UTC) Received: (qmail 87440 invoked by uid 500); 5 Aug 2011 16:34:29 -0000 Delivered-To: apmail-zookeeper-dev-archive@zookeeper.apache.org Received: (qmail 87240 invoked by uid 500); 5 Aug 2011 16:34:28 -0000 Mailing-List: contact dev-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list dev@zookeeper.apache.org Received: (qmail 87224 invoked by uid 99); 5 Aug 2011 16:34:28 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 16:34:28 +0000 Received: from localhost (HELO mail-yw0-f42.google.com) (127.0.0.1) (smtp-auth username phunt, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Aug 2011 16:34:28 +0000 Received: by ywb6 with SMTP id 6so1860724ywb.15 for ; Fri, 05 Aug 2011 09:34:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.42.146.10 with SMTP id h10mr2110143icv.300.1312562067120; Fri, 05 Aug 2011 09:34:27 -0700 (PDT) Received: by 10.42.108.134 with HTTP; Fri, 5 Aug 2011 09:34:27 -0700 (PDT) In-Reply-To: <69D3016305F9084FBD2C4A0DF189BD5C1770F90327@GSCMAMP02EX.firmwide.corp.gs.com> References:

<69D3016305F9084FBD2C4A0DF189BD5C1770F90303@GSCMAMP02EX.firmwide.corp.gs.com> <69D3016305F9084FBD2C4A0DF189BD5C1770D76002@GSCMAMP02EX.firmwide.corp.gs.com> <69D3016305F9084FBD2C4A0DF189BD5C1770F90327@GSCMAMP02EX.firmwide.corp.gs.com> Date: Fri, 5 Aug 2011 09:34:27 -0700 Message-ID: Subject: Re: devops/admin/client question: What do you do when you rollback? From: Patrick Hunt To: user@zookeeper.apache.org Cc: "dev@zookeeper.apache.org" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Aug 5, 2011 at 9:01 AM, Fournier, Camille F. wrote: > Actuallly.... can I update the ConnectRequest protocol version number? If= I can do that, I can have the server only send back the indicating Connect= Response on clients with a higher protocol version. It doesn't look like it= 's read anywhere right now. > (Moving this to dev since we've moved to a dev discussion) That's what I was going to suggest - upping the protocol version number for new clients. New servers can respond with ConnectionResponse2 if they see the new version, this response should have improved semantics. Otw they can just respond with the old version/resp. New clients will have to handle both types of responses. Patrick > > -----Original Message----- > From: Fournier, Camille F. [Tech] > Sent: Friday, August 05, 2011 11:57 AM > To: 'user@zookeeper.apache.org' > Subject: RE: devops/admin/client question: What do you do when you rollba= ck? > > Hmmm. I thought I had another way around this but I don't. We really didn= 't write the client to be easy to encode other errors in the connection res= ult... I think any good solution will have to be in our 4.0 clojure rewrite= ;) > > C > > > -----Original Message----- > From: Ted Dunning [mailto:ted.dunning@gmail.com] > Sent: Friday, August 05, 2011 11:51 AM > To: user@zookeeper.apache.org > Subject: Re: devops/admin/client question: What do you do when you rollba= ck? > > If you get the lower zxid from the leader then you know that things have > gone south. > > Likewise, if you get a lower epoch number from a node that thinks that it= is > in quorum then things are not good. =A0The definition of "thinks it is in > quorum" is problematic of course. > > On Fri, Aug 5, 2011 at 10:57 AM, Fournier, Camille F. < > Camille.Fournier@gs.com> wrote: > >> Oh blah, of course it won't be b/w compatible, because all the older >> clients would expire their sessions in the instance of a single zxid hig= her >> than the cluster zxid which I doubt most people want. >> >> Is there a way to check if the zxid of the client is higher than the >> current possible zxid after connection, and send the session_expired the= n? >> That would at least help us out most of the way. >> >> -----Original Message----- >> From: Patrick Hunt [mailto:phunt@apache.org] >> Sent: Thursday, August 04, 2011 7:23 PM >> To: user@zookeeper.apache.org >> Subject: Re: devops/admin/client question: What do you do when you >> rollback? >> >> Sounds reasonable to me as long as it's b/w compatible (which it seems >> like it would be), anything we can do to improve this situation would >> be huge - I frequently see our support team trying to address this >> (e.g. the max count exceeded issue) with clients like hbase. Def plus >> for supportability. >> >> Patrick >> >> On Thu, Aug 4, 2011 at 4:11 PM, Camille Fournier >> wrote: >> > I'm thinking of hacking it through the connectresponse session timeout >> > (similar to the way we detect session rejected). I wrote up a prototyp= e >> that >> > worked ok this way. Might could extend this hack to other things, usin= g >> that >> > field as an encoded error msg, thoughts? >> > >> > C >> > On Aug 4, 2011 6:10 PM, "Patrick Hunt" wrote: >> >> Our error reporting server->client has always been weak. It's a PITA >> >> to debug in production because a lot of times when the client gets >> >> bounced it's not clear from the client side why (you end up having to >> >> search the server log - for example when maxClientCount is exceeded). >> >> It would be great to fix this, esp if the server could provide insigh= t >> >> to the client about why (an error code/message perhaps). Doing it in = a >> >> b/w compatible way might be tough though... >> >> >> >> Patrick >> >> >> >> On Thu, Aug 4, 2011 at 2:45 PM, Ted Dunning >> wrote: >> >>> This is used normally to guarantee in-order data views. =A0If you ge= t >> >>> disconnected from one host in an advanced state and then connect to = an >> > out >> >>> of date slave, ZK automatically disconnects you to avoid letting you >> see >> >>> time go backwards. =A0Your situation is different of course. >> >>> >> >>> >> >>> >> >>> On Thu, Aug 4, 2011 at 7:05 PM, Fournier, Camille F. < >> >>> Camille.Fournier@gs.com> wrote: >> >>> >> >>>> Right now the server just detects that the zxid is wrong, and calls >> > close >> >>>> on the client. The client logs: >> >>>> 15:01:47,593 - INFO >> >>>> =A0[main-SendThread(localhost:2181):ClientCnxn$SendThread@1159] - >> Unable >> > to >> >>>> read additional data from server sessionid 0x131962b00540000, likel= y >> > server >> >>>> has closed socket, closing socket connection and attempting reconne= ct >> >>>> (branch 3.3.3) >> >>>> >> >>>> I will poke around and see if I can figure out a nicer way to indic= ate >> > this >> >>>> condition. The expired state is perfectly fine for me in my use cas= e. >> >>>> >> >>>> C >> >>>> >> >>>> >> >>>> -----Original Message----- >> >>>> From: Patrick Hunt [mailto:phunt@apache.org] >> >>>> Sent: Thursday, August 04, 2011 1:51 PM >> >>>> To: user@zookeeper.apache.org >> >>>> Subject: Re: devops/admin/client question: What do you do when you >> >>>> rollback? >> >>>> >> >>>> On Thu, Aug 4, 2011 at 10:29 AM, Fournier, Camille F. >> >>>> wrote: >> >>>> > We had an issue here the other day where the ZK servers were runn= ing >> >>>> poorly, and in an effort to get them healthy again we ended up roll= ing >> > back >> >>>> the cluster state. While this was, in retrospect, not the right >> solution >> > to >> >>>> the problem we were facing, it brought up another problem. Namely, >> that >> > many >> >>>> of our clients couldn't reconnect with their sessions because their >> zxid >> > was >> >>>> too high (expected), but that the error they got when trying to do >> that >> >>>> reconnection was just a vanilla disconnected error. The result was >> that >> > most >> >>>> of our clients had to be bounced. >> >>>> >> >>>> Hi Camille, there's a long standing jira on this: >> >>>> https://issues.apache.org/jira/browse/ZOOKEEPER-523 >> >>>> >> >>>> > Aside from trying hard to avoid ever rolling back the cluster sta= te, >> > does >> >>>> anyone have a way they deal with this situation if it occurs? Shoul= d >> we >> >>>> consider enhancing the error message to the client so we could trac= k >> the >> >>>> fact that we were ahead of the quorum zxid and react sensibly? >> > Alternately, >> >>>> since we were sending a sessionId along with the zxid, perhaps it >> would >> > be >> >>>> nice to check to see if the sessionId exists before checking the zx= id, >> > which >> >>>> would send an expired state signal which my client code could handl= e >> >>>> cleanly. >> >>>> >> >>>> It seems reasonable that if the client connects to all servers in t= he >> >>>> ensemble (that it knows about) and sees that it's ahead of each one= , >> >>>> it should consider the session expired (we could add a new state, b= ut >> >>>> seems like just treating as expired with a good log message would b= e >> >>>> better from b/w compat standpoint). >> >>>> >> >>>> I can't recall, does the client have sufficient information to make >> >>>> this determination, or is the server just disconnecting? >> >>>> >> >>>> Patrick >> >>>> >> >>> >> > >> >