hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahadev Konar <maha...@yahoo-inc.com>
Subject Re: rolling upgrade 3.2.1 -> 3.3.0
Date Wed, 14 Apr 2010 23:26:08 GMT
Hi Charity,
   Looks like you are hitting a bug recently found in 3.3.0.

https://issues.apache.org/jira/browse/ZOOKEEPER-737


Is the bug, wherein the server does not show the right status. Looks like in
your case the server is running fine but bin/zkserver.sh status is not
returning the right result.

You can try telnet localhost port and then type stat to get the status on
the server. This bug will be fixed in the bug fix release 3.3.1 which most
probalbly will be released by next week or so.

Thanks
mahadev
  


On 4/14/10 3:59 PM, "Charity Majors" <charity@shopkick.com> wrote:

> Hi.  I'm trying to upgrade a zookeeper cluster from 3.2.1 to 3.3.0, and having
> problems.  I can't get a 3.3.0 node to successfully join the cluster and stay
> joined.  
> 
> If I run zkServer.sh status immediately after starting up the newly upgraded
> node, it says the service is probably not running, and shows me this:
> 
> 
> [charity@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status
> JMX enabled by default
> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg
> 2010-04-14 22:47:35,574 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
> Accepted socket connection from /127.0.0.1:40287
> 2010-04-14 22:47:35,576 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Processing
> stat command from /127.0.0.1:40287
> 2010-04-14 22:47:35,577 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@606] -
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x0, likely client has closed socket
> 2010-04-14 22:47:35,578 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket
> connection for client /127.0.0.1:40287 (no session established for client)
> Error contacting service. It is probably not running.
> [charity@test-zookeeper001 zookeeper-current]$ 2010-04-14 22:47:35,580 - DEBUG
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1310] - ignoring
> exception during input shutdown
> java.net.SocketException: Transport endpoint is not connected
>         at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>         at 
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>         at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262)
> 
> 
> If I connect with zkCli.sh, I can list the contents of zookeeper.  If I make
> changes to the schema on either of the other two nodes, test-zookeeper002 and
> test-zookeeper003, both of which are running 3.2.1, the changes are reflected
> on test-zookeeper001, which is running 3.3.0.
> 
> When I exit zkCli.sh, however, zkServer.sh status starts flapping between
> "Error contacting service. It is probably not running." and "Mode: follower",
> as you can see below.
> 
> Any ideas?  I'd really rather not have to take the production zookeeper
> cluster down to upgrade if it's not necessary.
> 
> Thanks,
> Charity.
> 
> 
> 
> [charity@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status
> JMX enabled by default
> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg
> 2010-04-14 22:53:16,848 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
> Accepted socket connection from /127.0.0.1:55284
> 2010-04-14 22:53:16,849 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Processing
> stat command from /127.0.0.1:55284
> 2010-04-14 22:53:16,849 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@606] -
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x0, likely client has closed socket
> 2010-04-14 22:53:16,850 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket
> connection for client /127.0.0.1:55284 (no session established for client)
> Error contacting service. It is probably not running.
> 2010-04-14 22:53:16,850 - DEBUG
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1310] - ignoring
> exception during input shutdown
> java.net.SocketException: Transport endpoint is not connected
>         at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>         at 
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>         at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262)
> 
> [charity@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status
> JMX enabled by default
> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg
> 2010-04-14 22:53:18,908 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
> Accepted socket connection from /127.0.0.1:55285
> 2010-04-14 22:53:18,909 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Processing
> stat command from /127.0.0.1:55285
> 2010-04-14 22:53:18,909 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@606] -
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x0, likely client has closed socket
> 2010-04-14 22:53:18,910 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket
> connection for client /127.0.0.1:55285 (no session established for client)
> 2010-04-14 22:53:18,910 - DEBUG
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1310] - ignoring
> exception during input shutdown
> java.net.SocketException: Transport endpoint is not connected
>         at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>         at 
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>         at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262)
> 2010-04-14 22:53:18,911 - ERROR [Thread-13:NIOServerCnxn$Factory$1@82] -
> Thread Thread[Thread-13,5,main] died
> java.nio.channels.CancelledKeyException
>         at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
>         at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:64)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.wakeup(NIOServerCnx
> n.java:927)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.checkFlush(NIOServe
> rCnxn.java:909)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.flush(NIOServerCnxn
> .java:945)
>         at java.io.BufferedWriter.flush(BufferedWriter.java:236)
>         at java.io.PrintWriter.flush(PrintWriter.java:276)
>         at 
> org.apache.zookeeper.server.NIOServerCnxn$2.run(NIOServerCnxn.java:1089)
> Mode: follower
> 
> 


Mime
View raw message