Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Fri, 22 Apr 2016 14:24:12 +0000 (UTC)
From: "Josh Elser (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12961655.1461334995000.8859.1461335052895@Atlassian.JIRA>
In-Reply-To: <JIRA.12961655.1461334995000@Atlassian.JIRA>
References: <JIRA.12961655.1461334995000@Atlassian.JIRA>
 <JIRA.12961655.1461334995560@arcas>
Subject: [jira] [Commented] (ACCUMULO-4243) Kerberos thrift servers don't
 adhere to general.server.message.size.max
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15254023#comment-15254023 ] 

Josh Elser commented on ACCUMULO-4243:
--------------------------------------

I believe this is the corresponding stack trace for my telnet "client"

{noformat}
"ClientPool 259" daemon prio=5 tid=0x00007ff9de0d4800 nid=0xb75b runnable [0x0000000112075000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00000007493a4990> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
        at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
        at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
        at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
        at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - <0x00000007493a52d0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
{noformat}

> Kerberos thrift servers don't adhere to general.server.message.size.max
> -----------------------------------------------------------------------
>
>                 Key: ACCUMULO-4243
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4243
>             Project: Accumulo
>          Issue Type: Bug
>          Components: rpc
>    Affects Versions: 1.7.0, 1.7.1
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Blocker
>             Fix For: 1.7.2, 1.8.0
>
>
> It looks like when I implemented Kerberos client authentication in ACCUMULO-2815, I botched the use of general.server.message.size.max.
> This property is meant to guard us against trying to allocate a very large buffer from an RPC call. Typically, the first four bytes of data sent to one of our thrift servers (tserver, master, proxy) is treated as the number of bytes to read for some RPC. The problem is that if some garbage data (often, a security scan by some admins) may inadvertently cause the server to try to allocate a very large buffer (GBs in size) which will cause the process to ultimately crash while instantiating the buffer.
> It seems like something might be handled differently in the TSaslServer in Thrift. I'll have to dig more into whether it's an Accumulo or Thrift bug.
> To reproduce this, I was able to use telnet:
> {noformat}
> $ telnet `hostname -f` 9997
> <...waiting to get connected...>
> ~~~~000000000000000000
> ..
> {noformat}
> This will try to create a very large buffer (~2.1GB). I observed this by hooking up jvisualvm to the tabletserver.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)