hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charity Majors <char...@shopkick.com>
Subject Re: rolling upgrade 3.2.1 -> 3.3.0
Date Thu, 15 Apr 2010 00:09:58 GMT
Fantastic.  Thanks.


On Apr 14, 2010, at 4:26 PM, Mahadev Konar wrote:

> Hi Charity,
>   Looks like you are hitting a bug recently found in 3.3.0.
> 
> https://issues.apache.org/jira/browse/ZOOKEEPER-737
> 
> 
> Is the bug, wherein the server does not show the right status. Looks like in
> your case the server is running fine but bin/zkserver.sh status is not
> returning the right result.
> 
> You can try telnet localhost port and then type stat to get the status on
> the server. This bug will be fixed in the bug fix release 3.3.1 which most
> probalbly will be released by next week or so.
> 
> Thanks
> mahadev
> 
> 
> 
> On 4/14/10 3:59 PM, "Charity Majors" <charity@shopkick.com> wrote:
> 
>> Hi.  I'm trying to upgrade a zookeeper cluster from 3.2.1 to 3.3.0, and having
>> problems.  I can't get a 3.3.0 node to successfully join the cluster and stay
>> joined.  
>> 
>> If I run zkServer.sh status immediately after starting up the newly upgraded
>> node, it says the service is probably not running, and shows me this:
>> 
>> 
>> [charity@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status
>> JMX enabled by default
>> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg
>> 2010-04-14 22:47:35,574 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
>> Accepted socket connection from /127.0.0.1:40287
>> 2010-04-14 22:47:35,576 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Processing
>> stat command from /127.0.0.1:40287
>> 2010-04-14 22:47:35,577 - WARN
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@606] -
>> EndOfStreamException: Unable to read additional data from client sessionid
>> 0x0, likely client has closed socket
>> 2010-04-14 22:47:35,578 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket
>> connection for client /127.0.0.1:40287 (no session established for client)
>> Error contacting service. It is probably not running.
>> [charity@test-zookeeper001 zookeeper-current]$ 2010-04-14 22:47:35,580 - DEBUG
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1310] - ignoring
>> exception during input shutdown
>> java.net.SocketException: Transport endpoint is not connected
>>        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>        at 
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262)
>> 
>> 
>> If I connect with zkCli.sh, I can list the contents of zookeeper.  If I make
>> changes to the schema on either of the other two nodes, test-zookeeper002 and
>> test-zookeeper003, both of which are running 3.2.1, the changes are reflected
>> on test-zookeeper001, which is running 3.3.0.
>> 
>> When I exit zkCli.sh, however, zkServer.sh status starts flapping between
>> "Error contacting service. It is probably not running." and "Mode: follower",
>> as you can see below.
>> 
>> Any ideas?  I'd really rather not have to take the production zookeeper
>> cluster down to upgrade if it's not necessary.
>> 
>> Thanks,
>> Charity.
>> 
>> 
>> 
>> [charity@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status
>> JMX enabled by default
>> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg
>> 2010-04-14 22:53:16,848 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
>> Accepted socket connection from /127.0.0.1:55284
>> 2010-04-14 22:53:16,849 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Processing
>> stat command from /127.0.0.1:55284
>> 2010-04-14 22:53:16,849 - WARN
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@606] -
>> EndOfStreamException: Unable to read additional data from client sessionid
>> 0x0, likely client has closed socket
>> 2010-04-14 22:53:16,850 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket
>> connection for client /127.0.0.1:55284 (no session established for client)
>> Error contacting service. It is probably not running.
>> 2010-04-14 22:53:16,850 - DEBUG
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1310] - ignoring
>> exception during input shutdown
>> java.net.SocketException: Transport endpoint is not connected
>>        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>        at 
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262)
>> 
>> [charity@test-zookeeper001 zookeeper-current]$ bin/zkServer.sh status
>> JMX enabled by default
>> Using config: /services/zookeeper/zookeeper-20100412.1/bin/../conf/zoo.cfg
>> 2010-04-14 22:53:18,908 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] -
>> Accepted socket connection from /127.0.0.1:55285
>> 2010-04-14 22:53:18,909 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@968] - Processing
>> stat command from /127.0.0.1:55285
>> 2010-04-14 22:53:18,909 - WARN
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@606] -
>> EndOfStreamException: Unable to read additional data from client sessionid
>> 0x0, likely client has closed socket
>> 2010-04-14 22:53:18,910 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1286] - Closed socket
>> connection for client /127.0.0.1:55285 (no session established for client)
>> 2010-04-14 22:53:18,910 - DEBUG
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1310] - ignoring
>> exception during input shutdown
>> java.net.SocketException: Transport endpoint is not connected
>>        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>>        at 
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>>        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.closeSock(NIOServerCnxn.java:1306)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:1263)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:609)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:262)
>> 2010-04-14 22:53:18,911 - ERROR [Thread-13:NIOServerCnxn$Factory$1@82] -
>> Thread Thread[Thread-13,5,main] died
>> java.nio.channels.CancelledKeyException
>>        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
>>        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:64)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.wakeup(NIOServerCnx
>> n.java:927)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.checkFlush(NIOServe
>> rCnxn.java:909)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$SendBufferWriter.flush(NIOServerCnxn
>> .java:945)
>>        at java.io.BufferedWriter.flush(BufferedWriter.java:236)
>>        at java.io.PrintWriter.flush(PrintWriter.java:276)
>>        at 
>> org.apache.zookeeper.server.NIOServerCnxn$2.run(NIOServerCnxn.java:1089)
>> Mode: follower
>> 
>> 
> 


Mime
View raw message