impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Cox <william....@distilnetworks.com>
Subject Re: queries not being submitted in Impala cluster despite free resources
Date Wed, 01 Feb 2017 22:30:23 GMT
Thanks. We've removed HA Proxy to cause queries to admit to a single
coordinator and reduced the default query memory limit to about 5GB. Things
are looking pretty good. Y'all have been very helpful.
Thanks!
-William


On Wed, Feb 1, 2017 at 3:18 PM, Matthew Jacobs <mj@cloudera.com> wrote:

> Yes, your understanding is correct. This is a limitation of the
> current distributed design of the admission control mechanism, and one
> that we aim to improve by creating a centralized node for admission.
> In the meantime, you may need to avoid using a load balancer if you're
> sensitive to overadmission.
>
> On Wed, Feb 1, 2017 at 9:23 AM, William Cox
> <william.cox@distilnetworks.com> wrote:
> > Tim,
> >
> > I have a 7 node cluster with 159.71 GB available to each Impala node
> (1.1TB
> > available total) - the default resource allocation pool has 700GB
> allocated
> > - so 100GB per node.
> >
> > We have a "default query memory limit" set to 25GB. From reading
> > (https://www.cloudera.com/documentation/enterprise/5-8-
> x/topics/impala_mem_limit.html)
> > it would see that this means each node can only run 4 queries at once,
> since
> > Impala is requesting 25Gb per query regardless of the estimate (100/25 =
> 4).
>
> There is not a hard reservation yet, so the queries will consume as
> much as they can up to the mem limit. If there is 'overadmission',
> then more queries will begin executing and thus some may run oom.
>
> >
> > What I *don't* understand is how this works with running more than 4
> queries
> > *total* at any time - wouldn't Impala be asking for 25Gb for each query
> on
> > each node?
> >
> > It should also be noted that we set up HA proxy in front of Impala
> > (http://www.cloudera.com/documentation/enterprise/5-8-
> x/topics/impala_proxy.html)
> > because we have a lot of adhoc users. From reading the Admission Control
> > docs it seems that maybe that's part of the problem: "Note that admission
> > control currently offers only soft limits when multiple coordinators are
> > being used."
> >
> > So while I can only seem to run 4 queries per node, I can run more than 4
> > total because of the multiple coordinators?
> >
> > -William
> >
> >
> >
> > On Tue, Jan 31, 2017 at 2:08 PM, Tim Armstrong <tarmstrong@cloudera.com>
> > wrote:
> >>
> >> Do you have a default query memory limit set? Admission control does not
> >> generally work well if it's relying on the estimated memory requirement
> -
> >> you really need to have query memory limits set. If you have the default
> >> query memory limit set to 25GB, then admission control assumes that the
> >> query will use that amount on each node. I assume you mean 700GB memory
> >> total across all nodes - how much memory do you have per node?
> >>
> >> On Tue, Jan 31, 2017 at 7:31 AM, Jeszy <jeszyb@gmail.com> wrote:
> >>>
> >>> That would be good. If they eventually run successfully, a query
> profile
> >>> would also be welcome.
> >>>
> >>> Thanks
> >>>
> >>> On Tue, Jan 31, 2017 at 4:28 PM, William Cox
> >>> <william.cox@distilnetworks.com> wrote:
> >>>>
> >>>> Jeszy,
> >>>>
> >>>> Thanks for the suggestion. We also have a 25GB per-query limit set up.
> >>>> Queries that estimate a large size are rejected with an error stating
> they
> >>>> exceeded the memory limit. The queries I'm having trouble with are
> ones that
> >>>> have no such error but simply wait in the CREATED state. Next time it
> >>>> happens I'll see if I can grab the memory estimates and check.
> >>>> Thanks.
> >>>> -William
> >>>>
> >>>>
> >>>> On Tue, Jan 31, 2017 at 7:08 AM, Jeszy <jeszyb@gmail.com> wrote:
> >>>>>
> >>>>> Hey William,
> >>>>>
> >>>>> IIUC you have configured both a memory-based upper bound and a #
> >>>>> queries upper bound for the default pool. A query can get queued
if
> it would
> >>>>> exceed either of these limits. If you're not hitting the number
of
> queries
> >>>>> one, then it's probably memory, which can happen even if not fully
> utilized
> >>>>> - unless you specify a mem_limit for the query, the estimated memory
> >>>>> requirement will be used for deciding whether the query should be
> admitted.
> >>>>> This can get out of hand when the cardinality estimation is off,
> either due
> >>>>> to a very complex query or because of missing / old stats.
> >>>>>
> >>>>> This is about memory-based admission control exclusively, but I
think
> >>>>> it will be helpful:
> >>>>> http://www.cloudera.com/documentation/enterprise/
> latest/topics/impala_admission.html#admission_memory
> >>>>>
> >>>>> HTH
> >>>>>
> >>>>> On Mon, Jan 30, 2017 at 8:31 PM, William Cox
> >>>>> <william.cox@distilnetworks.com> wrote:
> >>>>>>
> >>>>>> I'm running CDH CDH-5.8.0-1 and Impala =version 2.6.0-cdh5.8.0
> RELEASE
> >>>>>> (build 8d8652f69461f0dd8d5f474573fb5de7ceb0ee6b). We have enabled
> resource
> >>>>>> management and allocated  ~700Gb of memory with 30 running queries
> for the
> >>>>>> default. Our background data jobs are Unlimited.
> >>>>>>
> >>>>>>
> >>>>>> In spite of this setup, we still encounter times where queries
will
> be
> >>>>>> marked as CREATED and waiting for allocation when the number
of
> running
> >>>>>> queries is well below 30 and the amount of used memory, as listed
> in the CDH
> >>>>>> UI, is well below 700GB.
> >>>>>>
> >>>>>> This is seemingly unpredicable. We've created extensive monitors
to
> >>>>>> track # of running queries and memory usage but there seems
to be
> no pattern
> >>>>>> to why/when these queries won't be submitted to the cluster.
> >>>>>>
> >>>>>> Is there some key metric that I might be missing or is there
any
> >>>>>> suggestions folks have for tracking down these queries that
won't be
> >>>>>> submitted?
> >>>>>> Thanks.
> >>>>>> -William
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>

Mime
View raw message