drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on
Date Thu, 08 Mar 2018 02:01:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390593#comment-16390593
] 

ASF GitHub Bot commented on DRILL-6187:
---------------------------------------

Github user sohami commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1145#discussion_r173041172
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlClient.java
---
    @@ -103,6 +96,23 @@ protected void handle(ControlConnection connection, int rpcType, ByteBuf
pBody,
         connection.getCurrentHandler().handle(connection, rpcType, pBody, dBody, sender);
       }
     
    +  @Override
    +  protected void prepareSaslHandshake(final RpcConnectionHandler<ControlConnection>
connectionListener)
    --- End diff --
    
    `RpcType.SASL_MESSAGE` message accessed within `prepareSaslHandshake` implementation of
DataClient/ControlClient is defined separately. Also each of these client except UserClient
has access to ConnectionConfig which is not part of BasicClient too and is used in prepareSaslHandshake
implementation. Hence I kept the implementations separate for both DataClient and ControlClient.


> Exception in RPC communication between DataClient/ControlClient and respective servers
when bit-to-bit security is on
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-6187
>                 URL: https://issues.apache.org/jira/browse/DRILL-6187
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - RPC, Security
>            Reporter: Sorabh Hamirwasia
>            Assignee: Sorabh Hamirwasia
>            Priority: Major
>             Fix For: 1.13.0
>
>
>  
> {color:#000000}Below is the summary of issue: {color}
>  
> {color:#000000}*Scenario:*{color}
> {color:#000000}It seems like first sendRecordBatch was sent to Foreman which initiated
the Authentication handshake. But before initiating handshake for auth we establish a connection
and store that in a registry. Now if in parallel there is another recordBatch (by a different
minor fragment running on same Drillbit) to be sent then that will see the connection available
in registry and will initiate the send. Before the authentication is completed this second
request reached foreman and it throws below exception saying RPC type 3 message is not allowed
and closes the connection. This also fails the authentication handshake which was in progress.{color}{color:#000000} Here
the logs with details:{color}
> {color:#000000} {color}
> {color:#000000}*Forman received the SASL_START message from another node:*{color}
> {color:#000000}*_2018-02-21 18:43:30,759 [_*{color}{color:#000000}_BitServer-4] TRACE
o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START from /10.10.100.161:35482_{color}
> {color:#000000} {color}
> {color:#000000}*Then around same time it received another message from client of Rpc
Type 3 which is for SendRecordBatch and fails since handshake is not completed yet.*{color}
> {color:#000000} {color}
> {color:#000000}*_2018-02-21 18:43:30,762_*{color}{color:#000000} _[BitServer-4] ERROR
o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /10.10.100.162:31012
<--> /__10.10.100.161:35482_ _(data server).  Closing connection._{color}
> {color:#000000}_io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException:
Request of type 3 is not allowed without authentication. Client on /__10.10.100.161:35482_
_must authenticate before making requests. Connection dropped. [Details: Encryption: enabled
, MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#000000} {color}
> {color:#000000}*Then client receives an channel closed exception:*{color}
> {color:#000000} {color}
> {color:#000000}*2018-02-21 18:43:30,764 [*{color}{color:#000000}BitClient-4] WARN  o.a.d.exec.rpc.RpcExceptionHandler
- Exception occurred with closed channel.  Connection: /_10.10.100.161:35482_ <-->
_10.10.100.162:31012_ (data client){color}
> {color:#000000} {color}
> {color:#000000}*and due to this it's initial command for authentication also fails. Since
there is channel closed exception above I will think that triggered the failure of authentication
request as well.*{color}
> {color:#000000} {color}
> {color:#000000}_Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while
establishing connection.  Failure type AUTHENTICATION._{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83)
~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataTunnel._{color}{color:#000000}*_sendRecordBatch_*{color}{color:#000000}_(DataTunnel.java:84)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.ops.AccountingDataTunnel.sendRecordBatch(AccountingDataTunnel.java:45)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000}        _at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:127)
~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#000000} {color}
> {color:#000000}So I think there is a concurrency issue where even though the authentication
is not completed the other requests are send to remote node as soon as TCP connection is available.
Instead it should wait until authentication is completed. Something like TCP connection should
be made available from registry only if authentication is completed.{color}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message