zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh R <rake...@huawei.com>
Subject RE: Re:RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
Date Fri, 28 Nov 2014 04:38:01 GMT
It was coming in one of the lab environment. Its a long run scenario and now its started running
with my patch mentioned earlier. Probably I'll add log when I get another chance.
It looks like the packet was not getting added to the 'pendingQueue' due to the following
condition.

                    sock.write(p.bb);
                    if (!p.bb.hasRemaining()) {
                        .....
                        .....
                        .....
                        pendingQueue.add(p);
                    }

-Rakesh

-----Original Message-----
From: Robin [mailto:rchzzjcn@163.com] 
Sent: 28 November 2014 07:13
To: user@zookeeper.apache.org
Cc: Zookeeper; zookeeper-user
Subject: Re:RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue,
but got "

Rakesh,


I believe your problem is a little  bit different than the bug I fired(ZOOKEEPER-2091). Your
problem is because the SASL Packe is directly sent out and not put into the pendingQueue,
so when the response Packet arrives, it will not find the request Packet in the pendingQueue.


BTW, could you add a log so that we can get more detail about the socket behavior ?
                    sock.write(p.bb);
                    while (p.bb.hasRemaining()) {
                        logger.info(...)
                        sock.write(p.bb);
                    }
Thanks,
- Robin








At 2014-11-27 20:09:35, "Rakesh R" <rakeshr@huawei.com> wrote:
>Yes exactly. I'm also thinking the same way. I have modified the code by iterating over
the buffer and write fully to the socket. 
>It looks fine now. Let me run the environment for some more time.
>
>                    sock.write(p.bb);
>                    while (p.bb.hasRemaining()) {
>                        sock.write(p.bb);
>                    }
>
>Regards,
>Rakesh
>-----Original Message-----
>From: Robin [mailto:rchzzjcn@163.com]
>Sent: 27 November 2014 17:34
>To: user@zookeeper.apache.org
>Cc: Zookeeper; zookeeper-user
>Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but
got "
>
>Hi Rakesh,
>
>By the following call flow:
>  SendThread#run()
>    -> ZooKeeperSaslClient#initialize()
>      -> ZooKeeperSaslClient#sendSaslPacket()
>        -> ClientCnxn#sendPacket()
>          -> SendThread#sendPacket()
>            -> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not
been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL
response comes, it will not find the corresponding Packet in the pendingQueue.
>
>- Robin
>
>
>At 2014-11-26 23:23:04, "Rakesh R" <rakeshr@huawei.com> wrote:
>>Hi,
>>
>>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper
client logs.
>>Client & server is using kerberoes based sasl authentication mechanism.
>>
>>I'm using 3.4.6 release.
>>
>>2014-11-25 22:06:19,114 INFO  [Thread-1-SendThread(vm-216:24002)]
>>Client will use GSSAPI as SASL mechanism. 
>>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslCli
>>e
>>nt.java:285)
>>2014-11-25 22:06:19,114 INFO  [Thread-1-SendThread(vm-216:24002)]
>>Opening socket connection to server vm-01/**.**.**.**:24002. Will 
>>attempt to SASL-authenticate using Login Context section 'Client'
>>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.
>>j
>>ava:1000)
>>2014-11-25 22:06:19,115 INFO  [Thread-1-SendThread(vm-216:24002)]
>>Socket connection established to vm-216/**.**.**.**:24002, initiating 
>>session 
>>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.
>>j
>>ava:855)
>>2014-11-25 22:06:19,116 INFO  [Thread-1-SendThread(vm-216:24002)]
>>Session establishment complete on server vm-216/**.**.**.**:24002, 
>>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>>1260)
>>2014-11-25 22:06:19,118 WARN  [Thread-1-SendThread(vm-216:24002)]
>>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002, 
>>unexpected error, closing socket connection and attempting reconnect
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>>java.io.IOException: Nothing in the queue, but got 425467
>>                at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
>>                at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
>>                at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
>>                at
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>>
>>
>>This exception has occured in the following logic.
>>
>>CilentCnxn#readResponse()
>>
>>            synchronized (pendingQueue) {
>>                if (pendingQueue.size() == 0) {
>>                    throw new IOException("Nothing in the queue, but got "
>>                            + replyHdr.getXid());
>>                }
>>                packet = pendingQueue.remove();
>>            }
>>
>>
>>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could
see is the following condition fails and missed to add it to the "pendingQueue"
>>
>>ClientCnxnSocketNIO.java
>>
>>                    sock.write(p.bb);
>>                    if (!p.bb.hasRemaining()) {
>>                        .....
>>                        .....
>>                        .....
>>                        pendingQueue.add(p);
>>                    }
>>
>>Does anyone phase similar issue and would appreciate any help. Thanks!
>>
>>Thanks & Regards,
>>Rakesh

Mime
View raw message