Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Wed, 26 Jul 2017 18:05:00 +0000 (UTC)
From: "Roman (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12970962.1463614669000.19973.1501092300306@Atlassian.JIRA>
In-Reply-To: <JIRA.12970962.1463614669000@Atlassian.JIRA>
References: <JIRA.12970962.1463614669000@Atlassian.JIRA> <JIRA.12970962.1463614669580@jira-lw-us.apache.org>
Subject: [jira] [Commented] (DRILL-4685) RpcException: Data not accepted
 downstream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 26 Jul 2017 18:10:00 -0000


    [ https://issues.apache.org/jira/browse/DRILL-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102035#comment-16102035 ] 

Roman commented on DRILL-4685:
------------------------------

I tried to reproduce this issue on 1 drillbit cluster with Drill from master and with Drill 1.7.0 using next scenarios:
- different types and sizes of tables (I tried on parquet and json tables, tried to increase size)
- memory allocation (tried different values of DRILL_MAX_DIRECT_MEMORY, DRILL_HEAP, planner.memory.max_query_memory_per_node)
- different number and sequence of the queries (I used up to 20 concurrent queries; tried step by step query 1 and query 2, concurrently run 10 queries 1 and 10 queries 2)

And I did not get "RpcException: Data not accepted downstream" in any case.

All I can get during running different scenarios are:

*1.* RESOURCE ERROR and ChannelClosedException on Drill 1.7.0

{code:xml}
[Error Id: ef88db17-f611-4d6e-85fa-22fee542ee01 on node1:31010]
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

Failure while allocating buffer.
Fragment 1:0

[Error Id: ef88db17-f611-4d6e-85fa-22fee542ee01 on node1:31010]
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:51)
	at ConcurrencyTest.selectData(ConcurrencyTest.java:37)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

Failure while allocating buffer.
Fragment 1:0

[Error Id: ef88db17-f611-4d6e-85fa-22fee542ee01 on node1:31010]
	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
	at oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:119)
	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
	at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:65)
	at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:363)
	at oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
	at oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:240)
	at oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:245)
	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
{code}

{code:xml}
[Error Id: 89f0891e-4dd9-49f1-a336-ff69af1302c4 on node1:31010]
java.sql.SQLException: SYSTEM ERROR: ChannelClosedException: Channel closed /192.168.121.7:41524 <--> /192.168.121.7:31012.

Fragment 2:1

[Error Id: 08208470-2620-46b1-a2b7-1b1ac619c714 on node1:31010]
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:51)
	at ConcurrencyTest.selectData(ConcurrencyTest.java:37)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
{code}

*2.* And CONNECTION ERROR on both of Drill.
{code:xml}
CONNECTION ERROR: Exceeded timeout (5000) while waiting send intermediate work fragments to remote nodes. Sent 1 and only heard response back from 0 nodes.


[Error Id: e51fa04d-c273-46da-b275-f2977a5c2aaf on node1:31010]
java.sql.SQLException: CONNECTION ERROR: Exceeded timeout (5000) while waiting send intermediate work fragments to remote nodes. Sent 1 and only heard response back from 0 nodes.


[Error Id: e51fa04d-c273-46da-b275-f2977a5c2aaf on node1:31010]
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
	at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1923)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:73)
	at oadd.net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
	at oadd.net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
	at oadd.net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
	at org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
	at ConcurrencyTesta.executeQuery(ConcurrencyTesta.java:49)
	at ConcurrencyTesta.selectData(ConcurrencyTesta.java:37)
	at ConcurrencyTesta.run(ConcurrencyTesta.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

{code}

But I think these errors I get according to small memory allocation (DRILL_MAX_DIRECT_MEMORY="6G", DRILL_HEAP="3G") and a big number of concurrency queries (my cluster so weak for this scenarios). So I can not reproduce "RpcException: Data not accepted downstream" exception.

> RpcException: Data not accepted downstream
> ------------------------------------------
>
>                 Key: DRILL-4685
>                 URL: https://issues.apache.org/jira/browse/DRILL-4685
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - RPC
>    Affects Versions: 1.7.0
>            Reporter: Rahul Challapalli
>            Assignee: Roman
>         Attachments: drillbit-receiver.log, drillbit-sender.log, instance1-profile.json, instance2-profile.json
>
>
> commit # : d93a3633815ed1c7efd6660eae62b7351a2c9739
> Scenario : Each of the below 2 queries are duplicated 10 times to make a total of 20 queries. Now I use 10 concurrent clients to run the queries.
> Query 1 : 
> {code}
> select count(*) from (
>   select max(length(concat(str1,str2))) max_length from (
>     select
>         substring(regexp_replace(s.enlarged_comment, 'iron..', 'iconic'), 4) str1,
>         substring(regexp_replace(s.enlarged_comment, 'm.*ne', 'iconic'), 4) str2
>     from (
>         select
>             concat(o_comment, o_comment, o_comment, o_comment, o_comment, o_comment,      o_comment, o_comment, o_comment, o_comment, o_comment, o_comment, o_comment,  o_comment, o_comment) enlarged_comment,
>             o_orderdate,
>             concat(o_clerk, '...') o_clerk
>         from orders_nocompression_256
>         where o_orderdate > date '1900-01-01' + interval '1' year
>       ) s
>       where
>           position('...' in o_clerk) > 0
>           and length(concat(enlarged_comment, o_clerk)) > 100
>       limit 5000000
>   ) s1
> )s2 where max_length = 0
> {code}
> Query 2 :
> {code}
> Select  count(*)  from lineitem_nocompression_256
> where
>     (
>         l_tax in (0.02,0.06,0.04,0.05,0.0,0.07,0.08,0.03,0.01)
>         and l_linestatus='F'
>     )
>     or (
>         ( length(l_comment) between 0 and 50)
>         and (
>             substr(l_shipmode, 1,2)='R'
>             or substr(l_shipmode, 1,2)='A'
>             and l_tax > 0.05
>         )
>     )
>     or (
>         (  l_extendedprice between 10000.0 and 100000.0 )
>         and l_linestatus='O'
>     )
>     or (
>         l_extendedprice*l_discount*l_tax < 45.00
>         and l_shipdate > date '1996-03-13' + interval '1' year
>     )
>     or (
>         l_commitdate in (
>             date '1996-02-12', date '1996-02-28', date '1996-03-05', date '1996-03-30', date '1996-03-14', date '1996-02-07', date '1997-01-14', date '1994-01-04'
>         )
>         and l_tax in (
>             0.02,0.06,0.04,0.05,0.0,0.07,0.08,0.03,0.01
>         )
>         and length(l_comment) > 15
>     )
>     or (
>         position('con' in regexp_replace(l_comment, 'm.*ne', 'iconic')) > 10
>         and (
>             length(regexp_replace(concat(lower(l_shipinstruct), lower(l_shipmode), l_comment), 'd.*ne', '')) > 0
>             or l_orderkey>50000
>             or l_partkey>1500
>             or l_linenumber=7
>         )
>     );
> {code}
> Out of the 20 queries submitted 2 copies of the first query failed with the same error. Below is the summary of the run
> {code}
> PASS (8.185 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 147182110)
> PASS (8.191 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 671870783)
> PASS (8.287 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 640915121)
> PASS (8.444 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 778960233)
> PASS (9.232 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 2022177583)
> PASS (2.423 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 778960233)
> PASS (11.67 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 1673732733)
> PASS (2.693 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1673732733)
> PASS (15.32 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1651684372)
> [#25] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: RpcException: Data not accepted downstream.
> Fragment 2:43
> [Error Id: 04168b77-dfdd-4e6c-9e86-33317c82947b on atsqa6c81.qa.lab:31010]
> 	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> 	at oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:114)
> 	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> 	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:65)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:363)
> 	at oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:240)
> 	at oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:245)
> 	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:744)
> [#26] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: RpcException: Data not accepted downstream.
> Fragment 2:11
> [Error Id: a155311b-3e50-44fe-94b5-2a258148ca4e on atsqa6c81.qa.lab:31010]
> 	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
> 	at oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:114)
> 	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:46)
> 	at oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:31)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:65)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$RequestEvent.run(RpcBus.java:363)
> 	at oadd.org.apache.drill.common.SerializedExecutor$RunnableProcessor.run(SerializedExecutor.java:89)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$SameExecutor.execute(RpcBus.java:240)
> 	at oadd.org.apache.drill.common.SerializedExecutor.execute(SerializedExecutor.java:123)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:245)
> 	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:744)
> EXECUTION_FAILURE (18.42 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 524239451)
> EXECUTION_FAILURE (18.43 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 297200484)
> PASS (12.53 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 671870783)
> PASS (9.851 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 778960233)
> PASS (20.73 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1166215941)
> PASS (3.144 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 524239451)
> PASS (16.74 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 147182110)
> PASS (11.74 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 1673732733)
> PASS (16.87 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 2022177583)
> PASS (17.82 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q1.q (connection: 640915121)
> PASS (10.98 min) /root/drillAutomation/framework/framework/resources/Advanced/concurrency/cpu_heavy/q2.q (connection: 1651684372)
> {code}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)