zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Reed <br...@yahoo-inc.com>
Subject Re: What does this mean?
Date Mon, 11 Oct 2010 17:16:23 GMT
  how big is your data? you may be running into the problem where it 
takes too long to do the state transfer and times out. check the 
initLimit and the size of your data.

ben

On 10/10/2010 08:57 AM, Avinash Lakshman wrote:
> Thanks Ben. I am not mixing processes of different clusters. I just double
> checked that. I have ZK deployed in a 5 node cluster and I have 20
> observers. I just started the 5 node cluster w/o starting the observers. I
> still the same issue. Now my cluster won't start up. So what is the correct
> workaround to get this going? How can I find out who the leader is and who
> the follower to get more insight?
>
> Thanks
> A
>
> On Sun, Oct 10, 2010 at 8:33 AM, Benjamin Reed<breed@yahoo-inc.com>  wrote:
>
>> this usually happens when a follower closes its connection to the leader.
>> it is usually caused by the follower shutting down or failing. you may get
>> further insight by looking at the follower logs. you should really run with
>> timestamps on so that you can correlate the logs of the leader and follower.
>>
>> on thing that is strange is the wide divergence between zxid of follower
>> and leader. are you mixing processes of different clusters?
>>
>> ben
>>
>> ________________________________________
>> From: Avinash Lakshman [avinash.lakshman@gmail.com]
>> Sent: Sunday, October 10, 2010 8:18 AM
>> To: zookeeper-user
>> Subject: What does this mean?
>>
>> I see this exception and the servers not doing anything.
>>
>> java.io.IOException: Channel eof
>>         at
>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
>> ERROR - 124554051584(higestZxid)>  21477836646(next log) for type -11
>> WARN - Sending snapshot last zxid of peer is 0xe00000000  zxid of leader is
>> 0x1e00000000
>> WARN - Sending snapshot last zxid of peer is 0x1800000000  zxid of leader
>> is
>> 0x1e00000000g
>>   WARN - Sending snapshot last zxid of peer is 0x5002dc766  zxid of leader
>> is
>> 0x1e00000000
>> WARN - Sending snapshot last zxid of peer is 0x1c00000000  zxid of leader
>> is
>> 0x1e00000000
>> ERROR - Unexpected exception causing shutdown while sock still open
>> java.net.SocketException: Broken pipe
>>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>>         at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>>         at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>>         at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:78)
>>         at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
>>         at
>> org.apache.jute.BinaryOutputArchive.writeInt(BinaryOutputArchive.java:55)
>>         at
>> org.apache.zookeeper.data.StatPersisted.serialize(StatPersisted.java:116)
>>         at org.apache.zookeeper.server.DataNode.serialize(DataNode.java:167)
>>         at
>>
>> org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
>>         at
>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:967)
>>         at
>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982)
>>         at
>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982)
>>         at
>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982)
>>         at
>> org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1031)
>>         at
>>
>> org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:104)
>>         at
>>
>> org.apache.zookeeper.server.ZKDatabase.serializeSnapshot(ZKDatabase.java:426)
>>         at
>>
>> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:331)
>> WARN - ******* GOODBYE /10.138.34.212:33272 ********
>>
>> Avinash
>>


Mime
View raw message