hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avinash Lakshman <avinash.laksh...@gmail.com>
Subject Re: What does this mean?
Date Mon, 11 Oct 2010 23:16:04 GMT
tickTime = 2000, initLimit = 3000 and the data is around 11GB this is log +
snapshot. So if I need to add a new observer can I transfer state from the
ensemble manually before starting it? If so which files do I need to
transfer?

Thanks

On Mon, Oct 11, 2010 at 10:16 AM, Benjamin Reed <breed@yahoo-inc.com> wrote:

>  how big is your data? you may be running into the problem where it takes
> too long to do the state transfer and times out. check the initLimit and the
> size of your data.
>
> ben
>
>
> On 10/10/2010 08:57 AM, Avinash Lakshman wrote:
>
>> Thanks Ben. I am not mixing processes of different clusters. I just double
>> checked that. I have ZK deployed in a 5 node cluster and I have 20
>> observers. I just started the 5 node cluster w/o starting the observers. I
>> still the same issue. Now my cluster won't start up. So what is the
>> correct
>> workaround to get this going? How can I find out who the leader is and who
>> the follower to get more insight?
>>
>> Thanks
>> A
>>
>> On Sun, Oct 10, 2010 at 8:33 AM, Benjamin Reed<breed@yahoo-inc.com>
>>  wrote:
>>
>>  this usually happens when a follower closes its connection to the leader.
>>> it is usually caused by the follower shutting down or failing. you may
>>> get
>>> further insight by looking at the follower logs. you should really run
>>> with
>>> timestamps on so that you can correlate the logs of the leader and
>>> follower.
>>>
>>> on thing that is strange is the wide divergence between zxid of follower
>>> and leader. are you mixing processes of different clusters?
>>>
>>> ben
>>>
>>> ________________________________________
>>> From: Avinash Lakshman [avinash.lakshman@gmail.com]
>>> Sent: Sunday, October 10, 2010 8:18 AM
>>> To: zookeeper-user
>>> Subject: What does this mean?
>>>
>>> I see this exception and the servers not doing anything.
>>>
>>> java.io.IOException: Channel eof
>>>        at
>>>
>>>
>>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
>>> ERROR - 124554051584(higestZxid)>  21477836646(next log) for type -11
>>> WARN - Sending snapshot last zxid of peer is 0xe00000000  zxid of leader
>>> is
>>> 0x1e00000000
>>> WARN - Sending snapshot last zxid of peer is 0x1800000000  zxid of leader
>>> is
>>> 0x1e00000000g
>>>  WARN - Sending snapshot last zxid of peer is 0x5002dc766  zxid of leader
>>> is
>>> 0x1e00000000
>>> WARN - Sending snapshot last zxid of peer is 0x1c00000000  zxid of leader
>>> is
>>> 0x1e00000000
>>> ERROR - Unexpected exception causing shutdown while sock still open
>>> java.net.SocketException: Broken pipe
>>>        at java.net.SocketOutputStream.socketWrite0(Native Method)
>>>        at
>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>>>        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>>>        at
>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>>>        at
>>> java.io.BufferedOutputStream.write(BufferedOutputStream.java:78)
>>>        at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
>>>        at
>>> org.apache.jute.BinaryOutputArchive.writeInt(BinaryOutputArchive.java:55)
>>>        at
>>> org.apache.zookeeper.data.StatPersisted.serialize(StatPersisted.java:116)
>>>        at
>>> org.apache.zookeeper.server.DataNode.serialize(DataNode.java:167)
>>>        at
>>>
>>>
>>> org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
>>>        at
>>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:967)
>>>        at
>>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982)
>>>        at
>>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982)
>>>        at
>>> org.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:982)
>>>        at
>>> org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1031)
>>>        at
>>>
>>>
>>> org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:104)
>>>        at
>>>
>>>
>>> org.apache.zookeeper.server.ZKDatabase.serializeSnapshot(ZKDatabase.java:426)
>>>        at
>>>
>>>
>>> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:331)
>>> WARN - ******* GOODBYE /10.138.34.212:33272 ********
>>>
>>> Avinash
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message