Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0F178200C54 for ; Wed, 29 Mar 2017 01:18:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0D815160B9C; Tue, 28 Mar 2017 23:18:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0590E160B9B for ; Wed, 29 Mar 2017 01:18:46 +0200 (CEST) Received: (qmail 99594 invoked by uid 500); 28 Mar 2017 23:18:46 -0000 Mailing-List: contact issues-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list issues@impala.incubator.apache.org Received: (qmail 99585 invoked by uid 99); 28 Mar 2017 23:18:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Mar 2017 23:18:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CEAA0183A68 for ; Tue, 28 Mar 2017 23:18:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id IqxJEPUvf0PC for ; Tue, 28 Mar 2017 23:18:43 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id D96F95F2FE for ; Tue, 28 Mar 2017 23:18:42 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 14043E039D for ; Tue, 28 Mar 2017 23:18:42 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id BDDF525CDD for ; Tue, 28 Mar 2017 23:18:41 +0000 (UTC) Date: Tue, 28 Mar 2017 23:18:41 +0000 (UTC) From: "Mostafa Mokhtar (JIRA)" To: issues@impala.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (IMPALA-5134) KRPC : Query with 2K fragments on un-secure 16 node cluster failed with ReportExecStatus RPC to 10.20.122.112:22000 timed out after 10.000s (ON_OUTBOUND_QUEUE) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 28 Mar 2017 23:18:48 -0000 Mostafa Mokhtar created IMPALA-5134: --------------------------------------- Summary: KRPC : Query with 2K fragments on un-secure 16 node c= luster failed with ReportExecStatus RPC to 10.20.122.112:22000 timed out af= ter 10.000s (ON_OUTBOUND_QUEUE) Key: IMPALA-5134 URL: https://issues.apache.org/jira/browse/IMPALA-5134 Project: IMPALA Issue Type: Sub-task Reporter: Mostafa Mokhtar Assignee: Henry Robinson Error message varies from run to run=20 EndDataStream RPC to 10.17.193.18:22000 timed out after 10.000s (SENT) EndDataStream RPC to 10.17.193.18:22000 timed out after 10.000s (ON_OUTBOUN= D_QUEUE) ExecPlanFragment RPC to 10.20.122.112:22000 timed out after 120.000s (SENT) Captured Vtune data while the query was running and noticed that the RPC ca= n spend significantl amount of time in tcmalloc which eventually spins in t= he kernel, this behavior can lead to unexpected RPC timeouts.=20 {code} CPU Time 2 of 101: 9.5% (2.435s of 25.740s) libc.so.6 ! madvise - [unknown source file] impalad ! TCMalloc_SystemRelease + 0x79 - [unknown source file] impalad ! tcmalloc::PageHeap::DecommitSpan + 0x20 - [unknown source file] impalad ! tcmalloc::PageHeap::MergeIntoFreeList + 0x212 - [unknown source f= ile] impalad ! tcmalloc::PageHeap::Delete + 0x23 - [unknown source file] impalad ! operator delete + 0x123 - [unknown source file] impalad ! ~faststring + 0x15 - faststring.h:54 impalad ! ~InboundTransfer - transfer.h:65 impalad ! kudu::DefaultDeleter::operator() - gs= coped_ptr.h:145 impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228 impalad ! ~gscoped_ptr - gscoped_ptr.h:318 impalad ! kudu::rpc::InboundCall::~InboundCall + 0xe7 - inbound_call.cc:51 impalad ! kudu::DefaultDeleter::operator() + 0x7 - = gscoped_ptr.h:145 impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228 impalad ! ~gscoped_ptr - gscoped_ptr.h:318 impalad ! ~ResponseTransferCallbacks + 0x30 - connection.cc:368 impalad ! ~ResponseTransferCallbacks - connection.cc:373 impalad ! kudu::rpc::ResponseTransferCallbacks::NotifyTransferFinished + 0x= 1e - connection.cc:376 impalad ! kudu::rpc::OutboundTransfer::SendBuffer + 0x1b9 - transfer.cc:221 impalad ! kudu::rpc::Connection::WriteHandler + 0x156 - connection.cc:596 impalad ! ev_invoke_pending + 0x52 - [unknown source file] impalad ! ev_run + 0x9c3 - [unknown source file] impalad ! ev::loop_ref::run + 0x12 - ev++.h:211 impalad ! kudu::rpc::ReactorThread::RunThread + 0x3 - reactor.cc:316 impalad ! boost::function0::operator() + 0x1a - function_template.hpp= :767 impalad ! kudu::Thread::SuperviseThread + 0x1ee - thread.cc:590 libpthread.so.0 ! start_thread + 0xd0 - [unknown source file] libc.so.6 ! clone + 0x6c - [unknown source file] {code} Query {code} select /* +straight_join */ count(*),a.c_nationkey, max(b.c_comment) from = customer A join /* +shuffle */ customer B on A.c_custkey =3D B.c_custkey = join /* +shuffle */ customer C on c.c_custkey =3D B.c_custkey join /* +sh= uffle */ customer D on d.c_custkey =3D B.c_custkey join /* +shuffle */ = customer E on e.c_custkey =3D B.c_custkey join /* +shuffle */ customer F = on f.c_custkey =3D B.c_custkey join /* +shuffle */ customer G on g.c_cust= key =3D B.c_custkey join /* +shuffle */ customer H on h.c_custkey =3D B.c= _custkey join /* +shuffle */ customer I on i.c_custkey =3D B.c_custkey jo= in /* +shuffle */ customer J on j.c_custkey =3D B.c_custkey join /* +shuf= fle */ customer K on k.c_custkey =3D B.c_custkey join /* +shuffle */ cu= stomer L on l.c_custkey =3D B.c_custkey join /* +shuffle */ customer M on= m.c_custkey =3D B.c_custkey join /* +shuffle */ customer N on n.c_custke= y =3D B.c_custkey join /* +shuffle */ customer O on o.c_custkey =3D B.c_c= ustkey join /* +shuffle */ customer P on p.c_custkey =3D B.c_custkey join= /* +shuffle */ customer R on R.c_custkey =3D B.c_custkey join /* +shuffl= e */ customer S on S.c_custkey =3D B.c_custkey join /* +shuffle */ cust= omer T on T.c_custkey =3D B.c_custkey join /* +shuffle */ customer U on U= .c_custkey =3D B.c_custkey join /* +shuffle */ customer V on V.c_custkey = =3D B.c_custkey join /* +shuffle */ customer W on W.c_custkey =3D B.c_cus= tkey join /* +shuffle */ customer X on X.c_custkey =3D B.c_custkey join /= * +shuffle */ customer Y on Y.c_custkey =3D B.c_custkey join /* +shuffle = */ customer Z on Z.c_custkey =3D B.c_custkey join /* +shuffle */ custom= er z1 on Z1.c_custkey =3D B.c_custkey join /* +shuffle */ customer z2 o= n Z2.c_custkey =3D B.c_custkey join /* +shuffle */ customer z3 on Z3.c_c= ustkey =3D B.c_custkey join /* +shuffle */ customer z4 on Z4.c_custkey = =3D B.c_custkey join /* +shuffle */ customer z5 on Z5.c_custkey =3D B.c_= custkey join /* +shuffle */ customer z6 on Z6.c_custkey =3D B.c_custkey = join /* +shuffle */ customer z7 on Z7.c_custkey =3D B.c_custkey join /* = +shuffle */ customer z8 on Z8.c_custkey =3D B.c_custkey join /* +shuffle= */ customer z9 on Z9.c_custkey =3D B.c_custkey join /* +shuffle */ cu= stomer z10 on Z10.c_custkey =3D B.c_custkey join /* +shuffle */ customer= z11 on Z11.c_custkey =3D B.c_custkey join /* +shuffle */ customer z12 = on Z12.c_custkey =3D B.c_custkey join /* +shuffle */ customer z13 on Z13= .c_custkey =3D B.c_custkey join /* +shuffle */ customer z14 on Z14.c_cus= tkey =3D B.c_custkey join /* +shuffle */ customer z15 on Z15.c_custkey = =3D B.c_custkey join /* +shuffle */ customer z16 on Z16.c_custkey =3D B.= c_custkey join /* +shuffle */ customer z17 on Z17.c_custkey =3D B.c_cust= key join /* +shuffle */ customer z18 on Z18.c_custkey =3D B.c_custkey jo= in /* +shuffle */ customer z19 on Z19.c_custkey =3D B.c_custkey join /* = +shuffle */ customer z20 on Z20.c_custkey =3D B.c_custkey join /* +shuff= le */ customer z21 on Z21.c_custkey =3D B.c_custkey join /* +shuffle */ = customer z22 on Z22.c_custkey =3D B.c_custkey join /* +shuffle */ cust= omer z23 on Z23.c_custkey =3D B.c_custkey join /* +shuffle */ customer z= 24 on Z24.c_custkey =3D B.c_custkey join /* +shuffle */ customer z25 on= Z25.c_custkey =3D B.c_custkey join /* +shuffle */ customer z26 on Z26.c= _custkey =3D B.c_custkey join /* +shuffle */ customer z27 on Z27.c_custk= ey =3D B.c_custkey join /* +shuffle */ customer z28 on Z28.c_custkey =3D= B.c_custkey join /* +shuffle */ customer z29 on Z29.c_custkey =3D B.c_c= ustkey join /* +shuffle */ customer z30 on Z30.c_custkey =3D B.c_custkey= join /* +shuffle */ customer z31 on Z31.c_custkey =3D B.c_custkey join = /* +shuffle */ customer z32 on Z32.c_custkey =3D B.c_custkey join /* +sh= uffle */ customer z33 on Z33.c_custkey =3D B.c_custkey join /* +shuffle = */ customer z34 on Z34.c_custkey =3D B.c_custkey join /* +shuffle */ c= ustomer z35 on Z35.c_custkey =3D B.c_custkey join /* +shuffle */ custome= r z36 on Z36.c_custkey =3D B.c_custkey join /* +shuffle */ customer z37 = on Z37.c_custkey =3D B.c_custkey join /* +shuffle */ customer z38 on Z3= 8.c_custkey =3D B.c_custkey join /* +shuffle */ customer z39 on Z39.c_cu= stkey =3D B.c_custkey join /* +shuffle */ customer z40 on Z40.c_custkey = =3D B.c_custkey join /* +shuffle */ customer z41 on Z41.c_custkey =3D B.= c_custkey join /* +shuffle */ customer z42 on Z42.c_custkey =3D B.c_cust= key join /* +shuffle */ customer z43 on Z43.c_custkey =3D B.c_custkey jo= in /* +shuffle */ customer z44 on z44.c_custkey =3D B.c_custkey join /* = +shuffle */ customer z45 on z45.c_custkey =3D B.c_custkey join /* +shuff= le */ customer z46 on z46.c_custkey =3D B.c_custkey join /* +shuffle */ = customer z47 on z47.c_custkey =3D B.c_custkey join /* +shuffle */ cust= omer z48 on z48.c_custkey =3D B.c_custkey join /* +shuffle */ customer z= 49 on z49.c_custkey =3D B.c_custkey join /* +shuffle */ customer z50 on= z50.c_custkey =3D B.c_custkey join /* +shuffle */ customer z51 on z51.c= _custkey =3D B.c_custkey join /* +shuffle */ customer z52 on z52.c_custk= ey =3D B.c_custkey join /* +shuffle */ customer z53 on z53.c_custkey =3D= B.c_custkey join /* +shuffle */ customer z54 on z54.c_custkey =3D B.c_c= ustkey join /* +shuffle */ customer z55 on z55.c_custkey =3D B.c_custkey= join /* +shuffle */ customer z56 on z56.c_custkey =3D B.c_custkey join = /* +shuffle */ customer z57 on z57.c_custkey =3D B.c_custkey join /* +sh= uffle */ customer z58 on z58.c_custkey =3D B.c_custkey join /* +shuffle = */ customer z59 on z59.c_custkey =3D B.c_custkey join /* +shuffle */ c= ustomer z60 on z60.c_custkey =3D B.c_custkey join /* +shuffle */ custome= r z61 on z61.c_custkey =3D B.c_custkey join /* +shuffle */ customer z62 = on z62.c_custkey =3D B.c_custkey join /* +shuffle */ customer z63 on z6= 3.c_custkey =3D B.c_custkey join /* +shuffle */ customer z64 on z64.c_cu= stkey =3D B.c_custkey join /* +shuffle */ customer z65 on z65.c_custkey = =3D B.c_custkey join /* +shuffle */ customer z66 on z66.c_custkey =3D B.= c_custkey join /* +shuffle */ customer z67 on z67.c_custkey =3D B.c_cust= key join /* +shuffle */ customer z68 on z68.c_custkey =3D B.c_custkey jo= in /* +shuffle */ customer z69 on z69.c_custkey =3D B.c_custkey join /* = +shuffle */ customer z70 on z70.c_custkey =3D B.c_custkey join /* +shuff= le */ customer z71 on z71.c_custkey =3D B.c_custkey join /* +shuffle */ = customer z72 on z72.c_custkey =3D B.c_custkey join /* +shuffle */ cust= omer z73 on z73.c_custkey =3D B.c_custkey join /* +shuffle */ customer z= 74 on z74.c_custkey =3D B.c_custkey join /* +shuffle */ customer z75 on= z75.c_custkey =3D B.c_custkey join /* +shuffle */ customer z76 on z76.c= _custkey =3D B.c_custkey join /* +shuffle */ customer z77 on z77.c_custk= ey =3D B.c_custkey join /* +shuffle */ customer z78 on z78.c_custkey =3D= B.c_custkey join /* +shuffle */ customer z79 on z79.c_custkey =3D B.c_c= ustkey join /* +shuffle */ customer z80 on z80.c_custkey =3D B.c_custkey= join /* +shuffle */ customer z81 on z81.c_custkey =3D B.c_custkey join = /* +shuffle */ customer z82 on z82.c_custkey =3D B.c_custkey join /* +sh= uffle */ customer z83 on z83.c_custkey =3D B.c_custkey join /* +shuffle = */ customer z84 on z84.c_custkey =3D B.c_custkey join /* +shuffle */ c= ustomer z85 on z85.c_custkey =3D B.c_custkey join /* +shuffle */ custome= r z86 on z86.c_custkey =3D B.c_custkey join /* +shuffle */ customer z87 = on z87.c_custkey =3D B.c_custkey join /* +shuffle */ customer z88 on z8= 8.c_custkey =3D B.c_custkey join /* +shuffle */ customer z89 on z89.c_cu= stkey =3D B.c_custkey join /* +shuffle */ customer z90 on z90.c_custkey = =3D B.c_custkey join /* +shuffle */ customer z91 on z91.c_custkey =3D B.= c_custkey join /* +shuffle */ customer z92 on z92.c_custkey =3D B.c_cust= key join /* +shuffle */ customer z93 on z93.c_custkey =3D B.c_custkey jo= in /* +shuffle */ customer z94 on z94.c_custkey =3D B.c_custkey join /* = +shuffle */ customer z95 on z95.c_custkey =3D B.c_custkey join /* +shuff= le */ customer z96 on z96.c_custkey =3D B.c_custkey join /* +shuffle */ = customer z97 on z97.c_custkey =3D B.c_custkey join /* +shuffle */ cust= omer z98 on z98.c_custkey =3D B.c_custkey join /* +shuffle */ customer z= 99 on z99.c_custkey =3D B.c_custkey join /* +shuffle */ customer z100 o= n z100.c_custkey =3D B.c_custkey and b.c_custkey < 10000 group by a.c_natio= nkey; {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)