Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EEC6C200C0E for ; Wed, 1 Feb 2017 23:30:30 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id ED4D8160B46; Wed, 1 Feb 2017 22:30:30 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C1837160B41 for ; Wed, 1 Feb 2017 23:30:29 +0100 (CET) Received: (qmail 91155 invoked by uid 500); 1 Feb 2017 22:30:29 -0000 Mailing-List: contact user-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@impala.incubator.apache.org Delivered-To: mailing list user@impala.incubator.apache.org Received: (qmail 91144 invoked by uid 99); 1 Feb 2017 22:30:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Feb 2017 22:30:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8AC0E185DBB for ; Wed, 1 Feb 2017 22:30:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.381 X-Spam-Level: ** X-Spam-Status: No, score=2.381 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=distilnetworks.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id OhZDeHUA2WxE for ; Wed, 1 Feb 2017 22:30:25 +0000 (UTC) Received: from mail-yw0-f177.google.com (mail-yw0-f177.google.com [209.85.161.177]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0369C5F19B for ; Wed, 1 Feb 2017 22:30:24 +0000 (UTC) Received: by mail-yw0-f177.google.com with SMTP id w75so76837104ywg.1 for ; Wed, 01 Feb 2017 14:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distilnetworks.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=GEdoSNoDkojk2Bo2KFbI1qVfkNybX36xHuTpxLQvbEQ=; b=SdyLhWJhXEvCbDAjn/gVEf74LzYKWDRBUmiCXBmNvpnIIlnNewtYDqWSCCJNlVdYdF HFAM1d17qY+hLAJ19UhoOFJQzd5nMjWaQL7mOo9eIYOS5VEBzrIqHpHlTKLfobGeeQ1u HBiI8G2r2Nlqac1cienVEKnOYKeY5m7XgRq/s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=GEdoSNoDkojk2Bo2KFbI1qVfkNybX36xHuTpxLQvbEQ=; b=cA2xf9s9n+kiGDTLD5fHPx9BTeTbWgDyPaSt1wyg2ZMhemM26d/Dg99yV9L7M9p5ox +52OipV80obGSx4Qhw2U9vBi09aW1eSlaBX49q2/w/CogIqUIFc7fV1BWvNqmKDpNBfF C6jOBAX1CU4ViYiQmG/3lueTScPLYZbMuaugOdGi5HYOeSdjRxf0PCeYFfy+UROpZM/b 6r4Rap7JV8f7EXNtqWbTIXgfzecRQNd9jeF/eY9v400d+LItx24o6eww0rMcIKhJrloU s8jFEnnlpIXFcmV7nWzJjtwIfSYgGqX+zLsEzP/M/YPpMk9A7KdbmSbu6xSfVbbkWvnB dTWg== X-Gm-Message-State: AIkVDXKbyg6NZ3DuVExPeNcakJvaNgxCi3zD6lLsWaQjANsZ33urZN/8a1mNFBFlIsnkKoIOJtleuHCQn/vGenFn X-Received: by 10.129.104.86 with SMTP id d83mr3974409ywc.244.1485988223892; Wed, 01 Feb 2017 14:30:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.104.131 with HTTP; Wed, 1 Feb 2017 14:30:23 -0800 (PST) In-Reply-To: References: From: William Cox Date: Wed, 1 Feb 2017 17:30:23 -0500 Message-ID: Subject: Re: queries not being submitted in Impala cluster despite free resources To: user@impala.incubator.apache.org Content-Type: multipart/alternative; boundary=001a1149087a3a1dfd05477f983d archived-at: Wed, 01 Feb 2017 22:30:31 -0000 --001a1149087a3a1dfd05477f983d Content-Type: text/plain; charset=UTF-8 Thanks. We've removed HA Proxy to cause queries to admit to a single coordinator and reduced the default query memory limit to about 5GB. Things are looking pretty good. Y'all have been very helpful. Thanks! -William On Wed, Feb 1, 2017 at 3:18 PM, Matthew Jacobs wrote: > Yes, your understanding is correct. This is a limitation of the > current distributed design of the admission control mechanism, and one > that we aim to improve by creating a centralized node for admission. > In the meantime, you may need to avoid using a load balancer if you're > sensitive to overadmission. > > On Wed, Feb 1, 2017 at 9:23 AM, William Cox > wrote: > > Tim, > > > > I have a 7 node cluster with 159.71 GB available to each Impala node > (1.1TB > > available total) - the default resource allocation pool has 700GB > allocated > > - so 100GB per node. > > > > We have a "default query memory limit" set to 25GB. From reading > > (https://www.cloudera.com/documentation/enterprise/5-8- > x/topics/impala_mem_limit.html) > > it would see that this means each node can only run 4 queries at once, > since > > Impala is requesting 25Gb per query regardless of the estimate (100/25 = > 4). > > There is not a hard reservation yet, so the queries will consume as > much as they can up to the mem limit. If there is 'overadmission', > then more queries will begin executing and thus some may run oom. > > > > > What I *don't* understand is how this works with running more than 4 > queries > > *total* at any time - wouldn't Impala be asking for 25Gb for each query > on > > each node? > > > > It should also be noted that we set up HA proxy in front of Impala > > (http://www.cloudera.com/documentation/enterprise/5-8- > x/topics/impala_proxy.html) > > because we have a lot of adhoc users. From reading the Admission Control > > docs it seems that maybe that's part of the problem: "Note that admission > > control currently offers only soft limits when multiple coordinators are > > being used." > > > > So while I can only seem to run 4 queries per node, I can run more than 4 > > total because of the multiple coordinators? > > > > -William > > > > > > > > On Tue, Jan 31, 2017 at 2:08 PM, Tim Armstrong > > wrote: > >> > >> Do you have a default query memory limit set? Admission control does not > >> generally work well if it's relying on the estimated memory requirement > - > >> you really need to have query memory limits set. If you have the default > >> query memory limit set to 25GB, then admission control assumes that the > >> query will use that amount on each node. I assume you mean 700GB memory > >> total across all nodes - how much memory do you have per node? > >> > >> On Tue, Jan 31, 2017 at 7:31 AM, Jeszy wrote: > >>> > >>> That would be good. If they eventually run successfully, a query > profile > >>> would also be welcome. > >>> > >>> Thanks > >>> > >>> On Tue, Jan 31, 2017 at 4:28 PM, William Cox > >>> wrote: > >>>> > >>>> Jeszy, > >>>> > >>>> Thanks for the suggestion. We also have a 25GB per-query limit set up. > >>>> Queries that estimate a large size are rejected with an error stating > they > >>>> exceeded the memory limit. The queries I'm having trouble with are > ones that > >>>> have no such error but simply wait in the CREATED state. Next time it > >>>> happens I'll see if I can grab the memory estimates and check. > >>>> Thanks. > >>>> -William > >>>> > >>>> > >>>> On Tue, Jan 31, 2017 at 7:08 AM, Jeszy wrote: > >>>>> > >>>>> Hey William, > >>>>> > >>>>> IIUC you have configured both a memory-based upper bound and a # > >>>>> queries upper bound for the default pool. A query can get queued if > it would > >>>>> exceed either of these limits. If you're not hitting the number of > queries > >>>>> one, then it's probably memory, which can happen even if not fully > utilized > >>>>> - unless you specify a mem_limit for the query, the estimated memory > >>>>> requirement will be used for deciding whether the query should be > admitted. > >>>>> This can get out of hand when the cardinality estimation is off, > either due > >>>>> to a very complex query or because of missing / old stats. > >>>>> > >>>>> This is about memory-based admission control exclusively, but I think > >>>>> it will be helpful: > >>>>> http://www.cloudera.com/documentation/enterprise/ > latest/topics/impala_admission.html#admission_memory > >>>>> > >>>>> HTH > >>>>> > >>>>> On Mon, Jan 30, 2017 at 8:31 PM, William Cox > >>>>> wrote: > >>>>>> > >>>>>> I'm running CDH CDH-5.8.0-1 and Impala =version 2.6.0-cdh5.8.0 > RELEASE > >>>>>> (build 8d8652f69461f0dd8d5f474573fb5de7ceb0ee6b). We have enabled > resource > >>>>>> management and allocated ~700Gb of memory with 30 running queries > for the > >>>>>> default. Our background data jobs are Unlimited. > >>>>>> > >>>>>> > >>>>>> In spite of this setup, we still encounter times where queries will > be > >>>>>> marked as CREATED and waiting for allocation when the number of > running > >>>>>> queries is well below 30 and the amount of used memory, as listed > in the CDH > >>>>>> UI, is well below 700GB. > >>>>>> > >>>>>> This is seemingly unpredicable. We've created extensive monitors to > >>>>>> track # of running queries and memory usage but there seems to be > no pattern > >>>>>> to why/when these queries won't be submitted to the cluster. > >>>>>> > >>>>>> Is there some key metric that I might be missing or is there any > >>>>>> suggestions folks have for tracking down these queries that won't be > >>>>>> submitted? > >>>>>> Thanks. > >>>>>> -William > >>>>>> > >>>>> > >>>> > >>> > >> > > > --001a1149087a3a1dfd05477f983d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks. We've removed HA Proxy to cause queries to adm= it to a single coordinator and reduced the=C2=A0default query memory limit = to about 5GB. Things are looking pretty good. Y'all have been very help= ful.
Thanks!
-William


On Wed, Feb 1, 2017 at 3:18 PM, Matthew = Jacobs <mj@cloudera.com> wrote:
Yes, your understanding is correct. This is a limitation of the
current distributed design of the admission control mechanism, and one
that we aim to improve by creating a centralized node for admission.
In the meantime, you may need to avoid using a load balancer if you're<= br> sensitive to overadmission.

On Wed, Feb 1, 2017 at 9:23 AM, William Cox
<william.cox@distilnet= works.com> wrote:
> Tim,
>
> I have a 7 node cluster with 159.71 GB available to each Impala node (= 1.1TB
> available total) - the default resource allocation pool has 700GB allo= cated
> - so 100GB per node.
>
> We have a "default query memory limit" set to 25GB. From rea= ding
> (https://ww= w.cloudera.com/documentation/enterprise/5-8-x/topics/impala_mem_l= imit.html)
> it would see that this means each node can only run 4 queries at once,= since
> Impala is requesting 25Gb per query regardless of the estimate (100/25= =3D 4).

There is not a hard reservation yet, so the queries will consume as<= br> much as they can up to the mem limit. If there is 'overadmission',<= br> then more queries will begin executing and thus some may run oom.

>
> What I *don't* understand is how this works with running more than= 4 queries
> *total* at any time - wouldn't Impala be asking for 25Gb for each = query on
> each node?
>
> It should also be noted that we set up HA proxy in front of Impala
> (http://www.clou= dera.com/documentation/enterprise/5-8-x/topics/impala_proxy.html<= /a>)
> because we have a lot of adhoc users. From reading the Admission Contr= ol
> docs it seems that maybe that's part of the problem: "Note th= at admission
> control currently offers only soft limits when multiple coordinators a= re
> being used."
>
> So while I can only seem to run 4 queries per node, I can run more tha= n 4
> total because of the multiple coordinators?
>
> -William
>
>
>
> On Tue, Jan 31, 2017 at 2:08 PM, Tim Armstrong <tarmstrong@cloudera.com>
> wrote:
>>
>> Do you have a default query memory limit set? Admission control do= es not
>> generally work well if it's relying on the estimated memory re= quirement -
>> you really need to have query memory limits set. If you have the d= efault
>> query memory limit set to 25GB, then admission control assumes tha= t the
>> query will use that amount on each node. I assume you mean 700GB m= emory
>> total across all nodes - how much memory do you have per node?
>>
>> On Tue, Jan 31, 2017 at 7:31 AM, Jeszy <jeszyb@gmail.com> wrote:
>>>
>>> That would be good. If they eventually run successfully, a que= ry profile
>>> would also be welcome.
>>>
>>> Thanks
>>>
>>> On Tue, Jan 31, 2017 at 4:28 PM, William Cox
>>> <william.= cox@distilnetworks.com> wrote:
>>>>
>>>> Jeszy,
>>>>
>>>> Thanks for the suggestion. We also have a 25GB per-query l= imit set up.
>>>> Queries that estimate a large size are rejected with an er= ror stating they
>>>> exceeded the memory limit. The queries I'm having trou= ble with are ones that
>>>> have no such error but simply wait in the CREATED state. N= ext time it
>>>> happens I'll see if I can grab the memory estimates an= d check.
>>>> Thanks.
>>>> -William
>>>>
>>>>
>>>> On Tue, Jan 31, 2017 at 7:08 AM, Jeszy <jeszyb@gmail.com> wrote:
>>>>>
>>>>> Hey William,
>>>>>
>>>>> IIUC you have configured both a memory-based upper bou= nd and a #
>>>>> queries upper bound for the default pool. A query can = get queued if it would
>>>>> exceed either of these limits. If you're not hitti= ng the number of queries
>>>>> one, then it's probably memory, which can happen e= ven if not fully utilized
>>>>> - unless you specify a mem_limit for the query, the es= timated memory
>>>>> requirement will be used for deciding whether the quer= y should be admitted.
>>>>> This can get out of hand when the cardinality estimati= on is off, either due
>>>>> to a very complex query or because of missing / old st= ats.
>>>>>
>>>>> This is about memory-based admission control exclusive= ly, but I think
>>>>> it will be helpful:
>>>>> http://www.cloudera.com/documentation/enterprise= /latest/topics/impala_admission.html#admission_memory >>>>>
>>>>> HTH
>>>>>
>>>>> On Mon, Jan 30, 2017 at 8:31 PM, William Cox
>>>>> <= william.cox@distilnetworks.com> wrote:
>>>>>>
>>>>>> I'm running CDH CDH-5.8.0-1 and Impala =3Dvers= ion 2.6.0-cdh5.8.0 RELEASE
>>>>>> (build 8d8652f69461f0dd8d5f474573fb5de7ceb0ee= 6b). We have enabled resource
>>>>>> management and allocated=C2=A0 ~700Gb of memory wi= th 30 running queries for the
>>>>>> default. Our background data jobs are Unlimited. >>>>>>
>>>>>>
>>>>>>= ; In spite of this setup, we still encounter times where queries will be >>>>>> marked as CREATED and waiting for allocation when = the number of running
>>>>>> queries is well below 30 and the amount of used me= mory, as listed in the CDH
>>>>>> UI, is well below 700GB.
>>>>>>
>>>>>> This is seemingly unpredicable. We've created = extensive monitors to
>>>>>> track # of running queries and memory usage but th= ere seems to be no pattern
>>>>>> to why/when these queries won't be submitted t= o the cluster.
>>>>>>
>>>>>> Is there some key metric that I might be missing o= r is there any
>>>>>> suggestions folks have for tracking down these que= ries that won't be
>>>>>> submitted?
>>>>>> Thanks.
>>>>>> -William
>>>>>>
>>>>>
>>>>
>>>
>>
>

--001a1149087a3a1dfd05477f983d--