Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 740EA200D0F for ; Fri, 29 Sep 2017 20:33:43 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 7271C1609D1; Fri, 29 Sep 2017 18:33:43 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 91DBD1609BC for ; Fri, 29 Sep 2017 20:33:42 +0200 (CEST) Received: (qmail 33304 invoked by uid 500); 29 Sep 2017 18:33:41 -0000 Mailing-List: contact user-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@impala.incubator.apache.org Delivered-To: mailing list user@impala.incubator.apache.org Received: (qmail 33294 invoked by uid 99); 29 Sep 2017 18:33:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Sep 2017 18:33:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 50C271A41E6 for ; Fri, 29 Sep 2017 18:33:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.401 X-Spam-Level: X-Spam-Status: No, score=-0.401 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id JH4jl19zKuxf for ; Fri, 29 Sep 2017 18:33:39 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id F0B595F29A for ; Fri, 29 Sep 2017 18:33:38 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id 126so267723vkj.9 for ; Fri, 29 Sep 2017 11:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=0nTlBsZwuXFyIfFlAqOcyZDQzi0/jZWusHLJfqd8hck=; b=cxea79z9FNsqBY2Wa4C0PzV42zdjuHos2s6R+HsP2cg8z0Joj5V+qhkjM83w5MdGLJ aifXNsg1EVVk2SpOdnmEatYYj/jRMuEA9dp7DQN7m97WCxxVezwpe5Z4e5zzihvAWvFH F40R02hklJmWs6dpDtrg3230LKSWV65UAhclWApJEAgAiDWcdO/WEya8MSQwWuXmTJol xDVAaq8bY/f7+JaMqdyAdWsL7I94ygUQDFG9ZLi0EcAlMm7ahbUdLo6oPGbSouCyU7Q3 buucwQRs3ffmenF5ebPZ1NDAs1OWuRO5O90Kd7OnNt5TgcTcJEnuEYEDHC95hTRMf5Y9 Vo7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=0nTlBsZwuXFyIfFlAqOcyZDQzi0/jZWusHLJfqd8hck=; b=C8BBwuB3+Njwuna4fWAc2CyTRrkFot7UV4mM96rBv7PKB65ZSQ0UbhIichNRGupRYE WPAD0SXHPZt5ezx9PqAh1yBoaM5YpRjyOL2xjQp+V8YC+nOdfbR2pRBvn10/rGZ1ZyH5 CZTBOz8kzSze69XJnhsNQOTDB7fM+8IqnZyMV/SAfTR6664Ceayr8OvCBdO7CFRzsoUH CJGp5q4CULP9n5tX7hFxLo9Y4QtHkmMYlIZCIi0ysKBuGq7WuQ1Hoqq7E2Di/alO0DjK LogcyW2JC+YfCdTdK8YPsToTjUhqdPo4HyLIZPhMoz7Uvfx9n+lwvXKvMSgKtFunvIXA 3ZMw== X-Gm-Message-State: AHPjjUg9mTgAgCrJkYk55h7QUcQVOq9bM+4KMhIbrXW0aMJZmM2+CwQ+ JhMhJucAwog1XeGaG1/VvO7Boa+qFJAI+cPAVUeaZRCi X-Google-Smtp-Source: AOwi7QBvXFh8zsaLmF4wCktG2BiB0od9YD4YnwyMYlXCzjE0WlyzjdutcVVbdOGFIXXzyQdlPJWv3e9CDF/uJ5+85Yw= X-Received: by 10.31.233.1 with SMTP id g1mr4543646vkh.62.1506710018304; Fri, 29 Sep 2017 11:33:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.174.72 with HTTP; Fri, 29 Sep 2017 11:33:07 -0700 (PDT) In-Reply-To: References: From: Tim Armstrong Date: Fri, 29 Sep 2017 11:33:07 -0700 Message-ID: Subject: Re: Impala Memory Leak (version 2.7.0-cdh5.10.2 ) To: user@impala.incubator.apache.org Content-Type: multipart/alternative; boundary="94eb2c0922fa6bf3f7055a5843ba" archived-at: Fri, 29 Sep 2017 18:33:43 -0000 --94eb2c0922fa6bf3f7055a5843ba Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I guess to elaborate further, there are two classes of scenarios: 1. the query hasn't been cancelled and closed on the coordinator so can't release resources. 2. the query has been closed on the coordinator but a fragment of the query is orphaned and still running. https://issues.apache.org/jira/browse/IMPALA-3633 is one possible cause of that that is fixed in 2.7.0. https://issues.apache.org/jira/browse/IMPALA-4925 is another possible case if the query had a limit. On Fri, Sep 29, 2017 at 11:30 AM, Tim Armstrong wrote: > It's not a memory leak, it's a query lifecycle issue. You have fragments > from two queries 2741ad08c43062bc:608bdb9500000000 and 8041bf903a5c7916:2= eba50b100000000 > that haven't been torn down. There are various ways that this could happe= n > but I don't know based on the information provided. > > I would check if those queries are still running and if you can cancel > them via Cloudera Manager or the Impala daemon web UI on the coordinator. > > On Fri, Sep 29, 2017 at 1:11 AM, Dong Bo =E8=91=A3=E5=8D=9A wrote: > >> Hi Experts, >> >> >> >> We are running queries on impalad version 2.7.0-cdh5.10.2 RELEASE (build= 38c989c0330ea952133111e41965ff9af96412d3), and issued with a memory leak p= roblem on the impala deamon . >> >> Once query end with error Memory limit exceeded , the allocated memory >> won=E2=80=99t be free until impala deamon restarted. Memory limit exce= eded is in >> attachement. >> >> >> >> I find jira tickets https://issues.apache.org/jira/browse/IMPALA-3589 , >> https://issues.apache.org/jira/browse/IMPALA-3582 with same issue as I >> am facing tagged as =E2=80=9CFixed=E2=80=9D in version 2.6.0, but strang= e I am still facing >> the issue in a higher version. >> >> >> >> Please kindly advise , it will be very helpful. >> >> >> >> Thanks & Regards >> >> Carl Dong >> >> >> >> >> >> >> > > --94eb2c0922fa6bf3f7055a5843ba Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I guess to elaborate further, there are two clas= ses of scenarios:
1. the query hasn't been cancelled and close= d on the coordinator so can't release resources.
2. the query = has been closed on the coordinator but a fragment of the query is orphaned = and still running. https://issues.apache.org/jira/browse/IMPALA-3633 is one possible= cause of that that is fixed in 2.7.0. https://issues.apache.org/jira/browse/IMPALA-4925= is another possible case if the query had a limit.

On Fri, Sep 29, 2017 at 11:= 30 AM, Tim Armstrong <tarmstrong@cloudera.com> wrote:<= br>
It's not a memo= ry leak, it's a query lifecycle issue. You have fragments from two quer= ies 2741ad08c43062bc:608bdb9500000000 and 8041bf903a5c7916:2eba50= b100000000 that haven't been torn down. There are various ways that thi= s could happen but I don't know based on the information provided.
<= br>
I would check if those queries are still running and if you can ca= ncel them via Cloudera Manager or the Impala daemon web UI on the coordinat= or.

On Fri, Sep 29, 2017 at 1:11 AM, Dong B= o =E8=91=A3=E5=8D=9A <dongbo@fosun.com> wrote:

Hi E= xperts,

=C2=A0
We are running queries on impalad version 2.7.0-cdh5.10.2 R=
ELEASE (build 38c989c0330ea952133111e41965ff9af96412d3), and issued wi=
th a memory leak problem on the impala deamon . =C2=A0=C2=A0=C2=A0=C2=A0 <=
/pre>

Once=
 query end with error Memory limit exceeded , the allocated memory won=E2=
=80=99t be free until impala deamon restarted. =C2=A0=C2=A0Memory limit exc=
eeded is in attachement.


<=
/u>=C2=A0


I fi=
nd jira tickets
https://issues.apache.org/jira/browse/IMPALA-3589 ,
https://issues.apache.org/jira/browse/IMPALA-3582 with same i=
ssue as I am facing tagged as =E2=80=9CFixed=E2=80=9D in version 2.6.0, but=
 strange I am still facing the issue in a higher version.


<=
/u>=C2=A0


Plea=
se kindly advise , it will be very helpful.


<=
/u>=C2=A0


Than=
ks & Regards


Carl=
 Dong


=C2=A0


=C2=A0


=C2=A0


--94eb2c0922fa6bf3f7055a5843ba--