zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-2775) ZK Client not able to connect with Xid out of order error
Date Tue, 16 May 2017 23:09:04 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013244#comment-16013244
] 

ASF GitHub Bot commented on ZOOKEEPER-2775:
-------------------------------------------

Github user hanm commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/254#discussion_r116879091
  
    --- Diff: src/java/test/org/apache/zookeeper/SaslAuthTest.java ---
    @@ -16,54 +16,58 @@
      * limitations under the License.
      */
     
    -package org.apache.zookeeper.test;
    +package org.apache.zookeeper;
    +
    +import static org.junit.Assert.assertTrue;
     
     import java.io.File;
     import java.io.FileWriter;
     import java.io.IOException;
    +import java.lang.reflect.Field;
     import java.util.ArrayList;
     import java.util.List;
     import java.util.concurrent.atomic.AtomicInteger;
     
    -import org.apache.zookeeper.CreateMode;
    -import org.apache.zookeeper.KeeperException;
    -import org.apache.zookeeper.TestableZooKeeper;
    -import org.apache.zookeeper.WatchedEvent;
    -import org.apache.zookeeper.ZooKeeper;
    +import org.apache.zookeeper.ClientCnxn.SendThread;
     import org.apache.zookeeper.Watcher.Event.KeeperState;
     import org.apache.zookeeper.ZooDefs.Ids;
     import org.apache.zookeeper.data.ACL;
     import org.apache.zookeeper.data.Id;
    +import org.apache.zookeeper.test.ClientBase;
    +import org.junit.AfterClass;
     import org.junit.Assert;
    +import org.junit.BeforeClass;
     import org.junit.Test;
     
     public class SaslAuthTest extends ClientBase {
    -    static {
    -        System.setProperty("zookeeper.authProvider.1","org.apache.zookeeper.server.auth.SASLAuthenticationProvider");
    -
    +    @BeforeClass
    +    public static void init() {
    +        System.setProperty("zookeeper.authProvider.1",
    +                "org.apache.zookeeper.server.auth.SASLAuthenticationProvider");
             try {
                 File tmpDir = createTmpDir();
                 File saslConfFile = new File(tmpDir, "jaas.conf");
                 FileWriter fwriter = new FileWriter(saslConfFile);
     
    -            fwriter.write("" +
    -                    "Server {\n" +
    -                    "          org.apache.zookeeper.server.auth.DigestLoginModule required\n"
+
    -                    "          user_super=\"test\";\n" +
    -                    "};\n" +
    -                    "Client {\n" +
    -                    "       org.apache.zookeeper.server.auth.DigestLoginModule required\n"
+
    -                    "       username=\"super\"\n" +
    -                    "       password=\"test\";\n" +
    -                    "};" + "\n");
    +            fwriter.write("" + "Server {\n"
    --- End diff --
    
    Nit: not sure if change is by accident or not, but this made readability worse.


> ZK Client not able to connect with Xid out of order error 
> ----------------------------------------------------------
>
>                 Key: ZOOKEEPER-2775
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2775
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: java client
>    Affects Versions: 3.4.10, 3.5.3, 3.6.0
>            Reporter: Bhupendra Kumar Jain
>            Assignee: Mohammad Arshad
>            Priority: Critical
>         Attachments: ZOOKEEPER-2775-01.patch
>
>
> During Network unreachable scenario in one of the cluster, we observed Xid out of order
and Nothing in the queue error continously. And ZK client it finally not able to connect successully
to ZK server. 
> *Logs:*
> unexpected error, closing socket connection and attempting reconnect | org.apache.zookeeper.ClientCnxn
(ClientCnxn.java:1447) 
> java.io.IOException: Xid out of order. Got Xid 52 with err 0 expected Xid 53 for a packet
with details: clientPath:null serverPath:null finished:false header:: 53,101  replyHeader::
0,0,-4  request:: 12885502275,v{'/app1/controller,'/app1/config/changes},v{},v{'/app1/config/changes}
 response:: null
> 	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:996)
> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101)
> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:370)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1426)
> unexpected error, closing socket connection and attempting reconnect 
> java.io.IOException: Nothing in the queue, but got 1
> 	at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:983)
> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101)
> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:370)
> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1426)
> 	
> *Analysis:* 
> 1) First time Client fails to do SASL login due to network unreachable problem.
> 2017-03-29 10:03:59,377 | WARN  | [main-SendThread(192.168.130.8:24002)] | SASL configuration
failed: javax.security.auth.login.LoginException: Network is unreachable (sendto failed) Will
continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows
it. | org.apache.zookeeper.ClientCnxn (ClientCnxn.java:1307) 
> 	Here the boolean saslLoginFailed becomes true.
> 2) After some time network connection is recovered and client is successully able to
login but still the boolean saslLoginFailed is not reset to false. 
> 3) Now SASL negotiation between client and server start happening and during this time
no user request will be sent. ( As the socket channel will be closed for write till sasl negotiation
complets)
> 4) Now response from server for SASL packet will be processed by the client and client
assumes that tunnelAuthInProgress() is finished ( method checks for saslLoginFailed boolean
Since the boolean is true it assumes its done.) and tries to process the packet as a other
packet and will result in above errors. 
> *Solution:*  Reset the saslLoginFailed boolean every time before client login



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message