zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Rosien <a...@rosien.net>
Subject Re: cluster member was switched to standalone, detectable?
Date Fri, 18 May 2012 22:21:29 GMT
Valid modes are: leader, follower, standalone, observer

If one node goes into standalone mode, your code will not detect a problem,
because standalone is a valid mode.

Monitoring like "ensure there is only one leader", "ensure no node is
standalone" are good things to have but require looking at all the nodes.

On Fri, May 18, 2012 at 1:57 PM, Jordan Zimmerman <jzimmerman@netflix.com>wrote:

> The 'srvr' command lists what mode the instance thinks it's in.
> Unfortunately, you have to manually parse it. If there's a quorum issue it
> outputs something like "This ZooKeeper is not serving requests".
>
> -JZ
>
> On 5/18/12 1:55 PM, "Adam Rosien" <adam@rosien.net> wrote:
>
> >Do the four-letter words tell me if a service joined the quorum correctly?
> >What commands and responses will tell me?
> >
> >How do I know what cluster it joined? What if nodes X & Y are in cluster A
> >but Z is in cluster B, should there be a cluster identifier to distinguish
> >membership?
> >
> >On Fri, May 18, 2012 at 12:05 PM, Patrick Hunt <phunt@apache.org> wrote:
> >
> >> That would detect it, I don't think it's avoidable in the sense that
> >> we can't detect that type of mis-configuration and somehow handle it
> >> (ie stop). Your best bet would be to automate the process (and test
> >> that ahead of time), or bring up the new server with the client port
> >> set to something previously unused, then verify, then restart it with
> >> the client port set as it was originally. I often do this when
> >> debugging issues. (but that itself might cause problems wrt config
> >> typos). Another option is to use iptables (etc...) to turn off access
> >> to clients until you've verified the server joined the quorum
> >> correctly, then turn off the filter.
> >>
> >> Patrick
> >>
> >> On Fri, May 18, 2012 at 11:51 AM, Jordan Zimmerman
> >> <jzimmerman@netflix.com> wrote:
> >> > ZooKeeper has a telnet style interface for periodic querying.
> >> >
> >> > You could also use Exhibitor and query it's REST API periodically. I
> >> > should probably add alerting to Exhibitor for this kind of thing.
> >> >
> >> > -JZ
> >> >
> >> > On 5/18/12 10:34 AM, "Adam Rosien" <adam@rosien.net> wrote:
> >> >
> >> >>We have a 5-member 3.3.3 cluster. One of the node's configurations was
> >> >>accidentally changed, and that node went into "standalone" mode,
> >>thinking
> >> >>it was a single-node cluster. However, all our zk clients still had
> >>the
> >> >>address of this server, and when connected obviously got missing or
> >>wrong
> >> >>data.
> >> >>
> >> >>Is this situation avoidable somehow?
> >> >>
> >> >>.. Adam
> >> >
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message