thrift-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stamatis Zampetakis (Jira)" <j...@apache.org>
Subject [jira] [Commented] (THRIFT-4598) TBinaryProtocol.readMessageBegin() hangs forever in Java
Date Tue, 22 Dec 2020 23:24:00 GMT

    [ https://issues.apache.org/jira/browse/THRIFT-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253801#comment-17253801
] 

Stamatis Zampetakis commented on THRIFT-4598:
---------------------------------------------

Apologies for commenting on a closed issue but I just encountered a very similar stacktrace
in some threads of Hiveserver2. 

They seem to hang for a long time in sockerRead0 while waiting to read the beginning of the
message. I don't have control over the client so cannot know if it is secure or insecure but
given that other requests by the same client are handled sucessfuly I am trying to understand
how do we end up in this situation where the thread hangs. Any other ideas apart from secure/insecure
client/server combination that might lead to the same behavior? 
{noformat}
java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
        - locked <0x0000000288af5280> (a java.lang.Object)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        - locked <0x0000000288af52c8> (a sun.security.ssl.AppInputStream)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x0000000283a4f9f0> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
        at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
        at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:706)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}


> TBinaryProtocol.readMessageBegin() hangs forever in Java
> --------------------------------------------------------
>
>                 Key: THRIFT-4598
>                 URL: https://issues.apache.org/jira/browse/THRIFT-4598
>             Project: Thrift
>          Issue Type: Bug
>          Components: Java - Library
>    Affects Versions: 0.11.0
>         Environment: Ubuntu Linux 16.04 x86_64
>            Reporter: Mario Emmenlauer
>            Assignee: James E. King III
>            Priority: Major
>
> I run some internal tests for the robustness of our thrift Java server. One of the robustness
tests is to try to connect to the secure server socket with an insecure client and vice versa.
This may seem a slightly diabolic test but it can be a simple user error to forget enabling
or disabling encryption. So I think thrift should handle this gracefully.
> However it seems that this test leads to an infinitely hanging server thread! The thread
blocks the jvm (Java 1.8.0_161 x86_64) from ending. I can see in jstack that the main method
ends and there are almost no threads left, except some garbage collectors and finalizers
plus org.apache.thrift.server.TThreadPoolServer.
> Here is the call stack of the thrift server thread:
> {code:java}
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x000000067b390fd8> (a java.io.BufferedInputStream)
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
> at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> I already reduced the server socket timeout to 30sec and the requestTimeout to 10sec.
Is there anything else I can do to timeout readMessageBegin()?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message