drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on
Date Thu, 08 Mar 2018 17:47:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391605#comment-16391605

ASF GitHub Bot commented on DRILL-6187:

Github user vrozov commented on a diff in the pull request:

    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
    @@ -371,17 +376,20 @@ protected void afterExecute(final Runnable r, final Throwable t)
         while (triedEndpointIndex < connectTriesVal) {
           endpoint = endpoints.get(triedEndpointIndex);
    +      // Set in both props and properties since props is passed to UserClient
    +      if (!properties.containsKey(DrillProperties.SERVICE_HOST)) {
    --- End diff --
    - Use `put` and add `TODO` comment.
     - What is a reason not to use API available in the minimum supported version assuming
that support for JDK 7 is dropped as part of https://issues.apache.org/jira/browse/DRILL-1491?

> Exception in RPC communication between DataClient/ControlClient and respective servers
when bit-to-bit security is on
> ---------------------------------------------------------------------------------------------------------------------
>                 Key: DRILL-6187
>                 URL: https://issues.apache.org/jira/browse/DRILL-6187
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - RPC, Security
>            Reporter: Sorabh Hamirwasia
>            Assignee: Sorabh Hamirwasia
>            Priority: Major
>             Fix For: 1.13.0
> {color:#000000}Below is the summary of issue: {color}
> {color:#000000}*Scenario:*{color}
> {color:#000000}It seems like first sendRecordBatch was sent to Foreman which initiated
the Authentication handshake. But before initiating handshake for auth we establish a connection
and store that in a registry. Now if in parallel there is another recordBatch (by a different
minor fragment running on same Drillbit) to be sent then that will see the connection available
in registry and will initiate the send. Before the authentication is completed this second
request reached foreman and it throws below exception saying RPC type 3 message is not allowed
and closes the connection. This also fails the authentication handshake which was in progress.{color}{color:#000000} Here
the logs with details:{color}
> {color:#000000} {color}
> {color:#000000}*Forman received the SASL_START message from another node:*{color}
> {color:#000000}*_2018-02-21 18:43:30,759 [_*{color}{color:#000000}_BitServer-4] TRACE
o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START from /{color}
> {color:#000000} {color}
> {color:#000000}*Then around same time it received another message from client of Rpc
Type 3 which is for SendRecordBatch and fails since handshake is not completed yet.*{color}
> {color:#000000} {color}
> {color:#000000}*_2018-02-21 18:43:30,762_*{color}{color:#000000} _[BitServer-4] ERROR
o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /
<--> /__10.10.100.161:35482_ _(data server).  Closing connection._{color}
> {color:#000000}_io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException:
Request of type 3 is not allowed without authentication. Client on /__10.10.100.161:35482_
_must authenticate before making requests. Connection dropped. [Details: Encryption: enabled
, MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#000000} {color}
> {color:#000000}*Then client receives an channel closed exception:*{color}
> {color:#000000} {color}
> {color:#000000}*2018-02-21 18:43:30,764 [*{color}{color:#000000}BitClient-4] WARN  o.a.d.exec.rpc.RpcExceptionHandler
- Exception occurred with closed channel.  Connection: /_10.10.100.161:35482_ <-->
_10.10.100.162:31012_ (data client){color}
> {color:#000000} {color}
> {color:#000000}*and due to this it's initial command for authentication also fails. Since
there is channel closed exception above I will think that triggered the failure of authentication
request as well.*{color}
> {color:#000000} {color}
> {color:#000000}_Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while
establishing connection.  Failure type AUTHENTICATION._{color}
> {color:#000000}        _at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
> {color:#000000}        _at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
> {color:#000000}        _at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
> {color:#000000}        _at org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83)
> {color:#000000}        _at org.apache.drill.exec.rpc.data.DataTunnel._{color}{color:#000000}*_sendRecordBatch_*{color}{color:#000000}_(DataTunnel.java:84)
> {color:#000000}        _at org.apache.drill.exec.ops.AccountingDataTunnel.sendRecordBatch(AccountingDataTunnel.java:45)
> {color:#000000}        _at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:127)
> {color:#000000} {color}
> {color:#000000}So I think there is a concurrency issue where even though the authentication
is not completed the other requests are send to remote node as soon as TCP connection is available.
Instead it should wait until authentication is completed. Something like TCP connection should
be made available from registry only if authentication is completed.{color}

This message was sent by Atlassian JIRA

View raw message