impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mostafa Mokhtar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IMPALA-5134) KRPC : Query with 2K fragments on un-secure 16 node cluster failed with ReportExecStatus RPC to 10.20.122.112:22000 timed out after 10.000s (ON_OUTBOUND_QUEUE)
Date Tue, 28 Mar 2017 23:18:41 GMT
Mostafa Mokhtar created IMPALA-5134:
---------------------------------------

             Summary: KRPC : Query with 2K fragments on un-secure 16 node cluster failed with
ReportExecStatus RPC to 10.20.122.112:22000 timed out after 10.000s (ON_OUTBOUND_QUEUE)
                 Key: IMPALA-5134
                 URL: https://issues.apache.org/jira/browse/IMPALA-5134
             Project: IMPALA
          Issue Type: Sub-task
            Reporter: Mostafa Mokhtar
            Assignee: Henry Robinson


Error message varies from run to run 
EndDataStream RPC to 10.17.193.18:22000 timed out after 10.000s (SENT)
EndDataStream RPC to 10.17.193.18:22000 timed out after 10.000s (ON_OUTBOUND_QUEUE)
ExecPlanFragment RPC to 10.20.122.112:22000 timed out after 120.000s (SENT)


Captured Vtune data while the query was running and noticed that the RPC can spend significantl
amount of time in tcmalloc which eventually spins in the kernel, this behavior can lead to
unexpected RPC timeouts. 

{code}
CPU Time
2 of 101: 9.5% (2.435s of 25.740s)

libc.so.6 ! madvise - [unknown source file]
impalad ! TCMalloc_SystemRelease + 0x79 - [unknown source file]
impalad ! tcmalloc::PageHeap::DecommitSpan + 0x20 - [unknown source file]
impalad ! tcmalloc::PageHeap::MergeIntoFreeList + 0x212 - [unknown source file]
impalad ! tcmalloc::PageHeap::Delete + 0x23 - [unknown source file]
impalad ! operator delete + 0x123 - [unknown source file]
impalad ! ~faststring + 0x15 - faststring.h:54
impalad ! ~InboundTransfer - transfer.h:65
impalad ! kudu::DefaultDeleter<kudu::rpc::InboundTransfer>::operator() - gscoped_ptr.h:145
impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
impalad ! ~gscoped_ptr - gscoped_ptr.h:318
impalad ! kudu::rpc::InboundCall::~InboundCall + 0xe7 - inbound_call.cc:51
impalad ! kudu::DefaultDeleter<kudu::rpc::InboundCall>::operator() + 0x7 - gscoped_ptr.h:145
impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
impalad ! ~gscoped_ptr - gscoped_ptr.h:318
impalad ! ~ResponseTransferCallbacks + 0x30 - connection.cc:368
impalad ! ~ResponseTransferCallbacks - connection.cc:373
impalad ! kudu::rpc::ResponseTransferCallbacks::NotifyTransferFinished + 0x1e - connection.cc:376
impalad ! kudu::rpc::OutboundTransfer::SendBuffer + 0x1b9 - transfer.cc:221
impalad ! kudu::rpc::Connection::WriteHandler + 0x156 - connection.cc:596
impalad ! ev_invoke_pending + 0x52 - [unknown source file]
impalad ! ev_run + 0x9c3 - [unknown source file]
impalad ! ev::loop_ref::run + 0x12 - ev++.h:211
impalad ! kudu::rpc::ReactorThread::RunThread + 0x3 - reactor.cc:316
impalad ! boost::function0<void>::operator() + 0x1a - function_template.hpp:767
impalad ! kudu::Thread::SuperviseThread + 0x1ee - thread.cc:590
libpthread.so.0 ! start_thread + 0xd0 - [unknown source file]
libc.so.6 ! clone + 0x6c - [unknown source file]
{code}


Query
{code}
select /* +straight_join */ count(*),a.c_nationkey, max(b.c_comment) from   customer A join
/* +shuffle */  customer B on A.c_custkey = B.c_custkey join /* +shuffle */   customer C on
c.c_custkey = B.c_custkey join /* +shuffle */   customer D on d.c_custkey = B.c_custkey join
/* +shuffle */   customer E on e.c_custkey = B.c_custkey join /* +shuffle */   customer F
on f.c_custkey = B.c_custkey join /* +shuffle */   customer G on g.c_custkey = B.c_custkey
join /* +shuffle */   customer H on h.c_custkey = B.c_custkey join /* +shuffle */   customer
I on i.c_custkey = B.c_custkey join /* +shuffle */   customer J on j.c_custkey = B.c_custkey
join /* +shuffle */   customer K on k.c_custkey = B.c_custkey join /* +shuffle */   customer
L on l.c_custkey = B.c_custkey join /* +shuffle */   customer M on m.c_custkey = B.c_custkey
join /* +shuffle */   customer N on n.c_custkey = B.c_custkey join /* +shuffle */   customer
O on o.c_custkey = B.c_custkey join /* +shuffle */   customer P on p.c_custkey = B.c_custkey
join /* +shuffle */   customer R on R.c_custkey = B.c_custkey join /* +shuffle */   customer
S on S.c_custkey = B.c_custkey join /* +shuffle */   customer T on T.c_custkey = B.c_custkey
join /* +shuffle */   customer U on U.c_custkey = B.c_custkey join /* +shuffle */   customer
V on V.c_custkey = B.c_custkey join /* +shuffle */   customer W on W.c_custkey = B.c_custkey
join /* +shuffle */   customer X on X.c_custkey = B.c_custkey join /* +shuffle */   customer
Y on Y.c_custkey = B.c_custkey join /* +shuffle */   customer Z on Z.c_custkey = B.c_custkey
join /* +shuffle */   customer z1  on Z1.c_custkey = B.c_custkey join /* +shuffle */   customer
z2  on Z2.c_custkey = B.c_custkey join /* +shuffle */   customer z3  on Z3.c_custkey = B.c_custkey
join /* +shuffle */   customer z4  on Z4.c_custkey = B.c_custkey join /* +shuffle */   customer
z5  on Z5.c_custkey = B.c_custkey join /* +shuffle */   customer z6  on Z6.c_custkey = B.c_custkey
join /* +shuffle */   customer z7  on Z7.c_custkey = B.c_custkey join /* +shuffle */   customer
z8  on Z8.c_custkey = B.c_custkey join /* +shuffle */   customer z9  on Z9.c_custkey = B.c_custkey
join /* +shuffle */   customer z10  on Z10.c_custkey = B.c_custkey join /* +shuffle */   customer
z11  on Z11.c_custkey = B.c_custkey join /* +shuffle */   customer z12  on Z12.c_custkey =
B.c_custkey join /* +shuffle */   customer z13  on Z13.c_custkey = B.c_custkey join /* +shuffle
*/   customer z14  on Z14.c_custkey = B.c_custkey join /* +shuffle */   customer z15  on Z15.c_custkey
= B.c_custkey join /* +shuffle */   customer z16  on Z16.c_custkey = B.c_custkey join /* +shuffle
*/   customer z17  on Z17.c_custkey = B.c_custkey join /* +shuffle */   customer z18  on Z18.c_custkey
= B.c_custkey join /* +shuffle */   customer z19  on Z19.c_custkey = B.c_custkey join /* +shuffle
*/   customer z20  on Z20.c_custkey = B.c_custkey join /* +shuffle */   customer z21  on Z21.c_custkey
= B.c_custkey join /* +shuffle */   customer z22  on Z22.c_custkey = B.c_custkey join /* +shuffle
*/   customer z23  on Z23.c_custkey = B.c_custkey join /* +shuffle */   customer z24  on Z24.c_custkey
= B.c_custkey join /* +shuffle */   customer z25  on Z25.c_custkey = B.c_custkey join /* +shuffle
*/   customer z26  on Z26.c_custkey = B.c_custkey join /* +shuffle */   customer z27  on Z27.c_custkey
= B.c_custkey join /* +shuffle */   customer z28  on Z28.c_custkey = B.c_custkey join /* +shuffle
*/   customer z29  on Z29.c_custkey = B.c_custkey join /* +shuffle */   customer z30  on Z30.c_custkey
= B.c_custkey join /* +shuffle */   customer z31  on Z31.c_custkey = B.c_custkey join /* +shuffle
*/   customer z32  on Z32.c_custkey = B.c_custkey join /* +shuffle */   customer z33  on Z33.c_custkey
= B.c_custkey join /* +shuffle */   customer z34  on Z34.c_custkey = B.c_custkey join /* +shuffle
*/   customer z35  on Z35.c_custkey = B.c_custkey join /* +shuffle */   customer z36  on Z36.c_custkey
= B.c_custkey join /* +shuffle */   customer z37  on Z37.c_custkey = B.c_custkey join /* +shuffle
*/   customer z38  on Z38.c_custkey = B.c_custkey join /* +shuffle */   customer z39  on Z39.c_custkey
= B.c_custkey join /* +shuffle */   customer z40  on Z40.c_custkey = B.c_custkey join /* +shuffle
*/   customer z41  on Z41.c_custkey = B.c_custkey join /* +shuffle */   customer z42  on Z42.c_custkey
= B.c_custkey join /* +shuffle */   customer z43  on Z43.c_custkey = B.c_custkey join /* +shuffle
*/   customer z44  on z44.c_custkey = B.c_custkey join /* +shuffle */   customer z45  on z45.c_custkey
= B.c_custkey join /* +shuffle */   customer z46  on z46.c_custkey = B.c_custkey join /* +shuffle
*/   customer z47  on z47.c_custkey = B.c_custkey join /* +shuffle */   customer z48  on z48.c_custkey
= B.c_custkey join /* +shuffle */   customer z49  on z49.c_custkey = B.c_custkey join /* +shuffle
*/   customer z50  on z50.c_custkey = B.c_custkey join /* +shuffle */   customer z51  on z51.c_custkey
= B.c_custkey join /* +shuffle */   customer z52  on z52.c_custkey = B.c_custkey join /* +shuffle
*/   customer z53  on z53.c_custkey = B.c_custkey join /* +shuffle */   customer z54  on z54.c_custkey
= B.c_custkey join /* +shuffle */   customer z55  on z55.c_custkey = B.c_custkey join /* +shuffle
*/   customer z56  on z56.c_custkey = B.c_custkey join /* +shuffle */   customer z57  on z57.c_custkey
= B.c_custkey join /* +shuffle */   customer z58  on z58.c_custkey = B.c_custkey join /* +shuffle
*/   customer z59  on z59.c_custkey = B.c_custkey join /* +shuffle */   customer z60  on z60.c_custkey
= B.c_custkey join /* +shuffle */   customer z61  on z61.c_custkey = B.c_custkey join /* +shuffle
*/   customer z62  on z62.c_custkey = B.c_custkey join /* +shuffle */   customer z63  on z63.c_custkey
= B.c_custkey join /* +shuffle */   customer z64  on z64.c_custkey = B.c_custkey join /* +shuffle
*/   customer z65  on z65.c_custkey = B.c_custkey join /* +shuffle */   customer z66  on z66.c_custkey
= B.c_custkey join /* +shuffle */   customer z67  on z67.c_custkey = B.c_custkey join /* +shuffle
*/   customer z68  on z68.c_custkey = B.c_custkey join /* +shuffle */   customer z69  on z69.c_custkey
= B.c_custkey join /* +shuffle */   customer z70  on z70.c_custkey = B.c_custkey join /* +shuffle
*/   customer z71  on z71.c_custkey = B.c_custkey join /* +shuffle */   customer z72  on z72.c_custkey
= B.c_custkey join /* +shuffle */   customer z73  on z73.c_custkey = B.c_custkey join /* +shuffle
*/   customer z74  on z74.c_custkey = B.c_custkey join /* +shuffle */   customer z75  on z75.c_custkey
= B.c_custkey join /* +shuffle */   customer z76  on z76.c_custkey = B.c_custkey join /* +shuffle
*/   customer z77  on z77.c_custkey = B.c_custkey join /* +shuffle */   customer z78  on z78.c_custkey
= B.c_custkey join /* +shuffle */   customer z79  on z79.c_custkey = B.c_custkey join /* +shuffle
*/   customer z80  on z80.c_custkey = B.c_custkey join /* +shuffle */   customer z81  on z81.c_custkey
= B.c_custkey join /* +shuffle */   customer z82  on z82.c_custkey = B.c_custkey join /* +shuffle
*/   customer z83  on z83.c_custkey = B.c_custkey join /* +shuffle */   customer z84  on z84.c_custkey
= B.c_custkey join /* +shuffle */   customer z85  on z85.c_custkey = B.c_custkey join /* +shuffle
*/   customer z86  on z86.c_custkey = B.c_custkey join /* +shuffle */   customer z87  on z87.c_custkey
= B.c_custkey join /* +shuffle */   customer z88  on z88.c_custkey = B.c_custkey join /* +shuffle
*/   customer z89  on z89.c_custkey = B.c_custkey join /* +shuffle */   customer z90  on z90.c_custkey
= B.c_custkey join /* +shuffle */   customer z91  on z91.c_custkey = B.c_custkey join /* +shuffle
*/   customer z92  on z92.c_custkey = B.c_custkey join /* +shuffle */   customer z93  on z93.c_custkey
= B.c_custkey join /* +shuffle */   customer z94  on z94.c_custkey = B.c_custkey join /* +shuffle
*/   customer z95  on z95.c_custkey = B.c_custkey join /* +shuffle */   customer z96  on z96.c_custkey
= B.c_custkey join /* +shuffle */   customer z97  on z97.c_custkey = B.c_custkey join /* +shuffle
*/   customer z98  on z98.c_custkey = B.c_custkey join /* +shuffle */   customer z99  on z99.c_custkey
= B.c_custkey join /* +shuffle */   customer z100  on z100.c_custkey = B.c_custkey and b.c_custkey
< 10000 group by a.c_nationkey;
{code}






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message