impala-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Jacobs>
Subject Re: Impala user resource isolation best practice
Date Tue, 26 Jul 2016 05:39:55 GMT
If memory cannot be spilled to disk then the query will be killed.


On Mon, Jul 25, 2016 at 7:26 PM, 廖松博 <> wrote:
> Hi Matt,
>         You mentioned that " If the query tries to use more than the mem limit it will
spill if possible" , if not possible what happened, the query will be killed or any unexpected
result ?
>         Thanks.
> Songbo
> -----邮件原件-----
> 发件人: Matthew Jacobs []
> 发送时间: 2016年7月26日 8:33
> 收件人: 廖松博
> 抄送:
> 主题: Re: Impala user resource isolation best practice
> On Mon, Jul 25, 2016 at 4:36 PM, 廖松博 <> wrote:
>> Hi Matt,
>>         Thanks for reply, but there are still something confuse me:
>>         1. Per
mem_limit is the maximum amount of memory a query can allocate on each node, not across the
entire cluster, right ? If we set mem limit, the estimation does not work ?
> Basically yes. You don't want to rely on estimates because they're going to be wrong.
>>         2. So if we set query mem limit = 5G, the coordinator will check if all the
workers has at least 5G memory , if not it will reject the query, right?
> Yes, partially. Right now it's not smart enough to actually reserve memory, but if it
thinks it the resources are available it will let the query start. If the coordinator thinks
the resources are already being used, the query will queue. If there isn't enough memory for
Impala on the node, then yes, the query will be rejected.
>>         3. If the coordinator accept the query and dispatch to workers, what will
happen if the query use 10G (more than mem_limit we set ) on that node during execution?Be
canceled or it is possible out of memory to crash the impalad process ? Does it use estimation
to do anything during execution ? Or estimation is not useful when we set query mem limit
> If the query tries to use more than the mem limit it will spill if possible. If it can't
spill, the query will be cancelled. It should not crash -- that would be a bug.  Estimates
are not used at runtime.
>>         4. If we does not set query mem limit, coordinator will use its estimation
instead of query limit to check if workers have enough memory to execute, right?
> You probably won't get what you expect. The estimates will be very far off and there
are no mem limits enforced, so all queries will just start consuming whatever they want.
>>         Thanks a lot for your patient
>> Songbo
>> -----邮件原件-----
>> 发件人: Matthew Jacobs []
>> 发送时间: 2016年7月26日 0:24
>> 收件人: 廖松博 <>
>> 抄送:
>> 主题: Re: Impala user resource isolation best practice
>> Hi,
>> If you set the query mem_limit the estimates are not used in this way, and they're
enforced at runtime. We recommend that you always set the query mem_limits. In Impala 2.5
you can configure the admission control pools to have a default query option mem_limit, e.g.
every query in poolA gets a default query mem_limit=5G, and perhaps poolB gets a default query
mem_limit=10g. You can still override those query options if you'd like with "set mem_limit=X;".
There is also an impalad command line argument to set the default query options (fallback
in all cases): --default_query_options=‘mem_limit=X’
>> Best,
>> Matt
>> On Sun, Jul 24, 2016 at 8:25 PM, 廖松博 <> wrote:
>>> Hi Matt,
>>>         Thanks for suggestion. Even though we can solve "soft limit" problem
using single coordinator, the coordinator doesn't know how much memory will be used during
execution. The limit is actually based on the memory size estimated by coordinator, right.
So it is possible that the coordinator accept a query which using estimating 5G memory which
under limit and dispatch to workers, but the query actually use 15G memory during execution
so limit doesn’t make sense.
>>>         And as I know the estimation will be more accurate if the table has stats,
right ? But it does not promise to be accurate even if stats exists.
>>>         Does the wrongly estimation matters ? For example will it cause the out
of memory problem ? If yes, is there any best practice to avoid that ?
>>>         Thank you very much.
>>> Songbo
>>> -----邮件原件-----
>>> 发件人: Matthew Jacobs []
>>> 发送时间: 2016年7月19日 11:35
>>> 收件人: 廖松博
>>> 抄送:
>>> 主题: Re: Impala user resource isolation best practice
>>> Hi Songbo,
>>> Yes, setting the MEM_LIMITs is a good idea. In Impala 2.5 and after, you can
set a per-pool default query mem limit. Cloudera Manager makes this easy in the Dynamic Resource
Pools page.
>>> You are right about the possibility of overwhelming a single coordinator. You
can consider setting up one larger node to be a coordinator (e.g. a lot of RAM & CPUs),
and setting it so that it *does not* have a datanode on it to minimize the number of query
fragments scheduled on it. In order to do that you'll also need to set the impalad cmd line
argument "-abort_on_config_error=false" because it normally expects a DN to be colocated with
the Impalad. It is still possible to overwhelm a single coordinator like this, but we've found
it to work reasonably well in our tests. Of course it is very dependent on your hardware and
>>> Best,
>>> Matt
>>> On Mon, Jul 18, 2016 at 6:31 PM, 廖松博 <> wrote:
>>>> Hi Matt,
>>>>         If we use a single coordinator, if a lot of queries come in, the
coordinator may be out of memory, right ? Because the coordinator take charge of merge results
from workers and send back to client.
>>>>         Thanks.
>>>> Songbo
>>>> -----邮件原件-----
>>>> 发件人: Matthew Jacobs []
>>>> 发送时间: 2016年7月19日 7:50
>>>> 收件人: 廖松博
>>>> 抄送:
>>>> 主题: Re: Impala user resource isolation best practice
>>>> The 'soft limits' refer to the behavior that, when multiple coordinators
are used and queries are submitted concurrently, the admission control cannot guarantee the
limits because the decisions are made based on some stale information that needs to be updated
by the statestore. If you use a single coordinator you avoid that problem. That said, a separate
problem is that there is no way to guarantee that other resources, e.g. network, IO, etc.
are not abused by a single user.
>>>> On Mon, Jul 18, 2016 at 4:29 PM, 廖松博 <>
>>>>> Hi Matthew,
>>>>>         Thanks for your reply. The point is , per admission control documents,
most of impala limits are "soft limit", are the 2 settings you mentioned also "soft limit"
? The soft limit means the pool will exceed the memory/concurrency limit at some moment when
impala is not aware of. But it is affecting other pool at that moment.
>>>>>         Thanks.
>>>>> Songbo
>>>>> -----邮件原件-----
>>>>> 发件人: Matthew Jacobs []
>>>>> 发送时间: 2016年7月19日 0:44
>>>>> 收件人:
>>>>> 主题: Re: Impala user resource isolation best practice
>>>>> By the way, some of the controls I mentioned were added in Impala 2.5,
so you should consider upgrading if you're not already using a newer version of Impala.
>>>>> Thanks,
>>>>> Matt
>>>>> On Mon, Jul 18, 2016 at 9:20 AM, Matthew Jacobs <>
>>>>>> Hi Songbo,
>>>>>> Right now the best you can do is with admission control with:
>>>>>> (a) a single coordinator to avoid the possibility of
>>>>>> over-admitting by different coordinators
>>>>>> (b) setting default query mem limits so that individual queries
>>>>>> are limited
>>>>>> For your scenario, I'd recommend setting up 2 pools, one for user
>>>>>> A and a second for user B. Set the max number of running queries
>>>>>> for user A to something reasonable for the concurrency for that workload.
>>>>>> Set the max memory for the user B pool to the portion of cluster
>>>>>> memory you're willing to give to those queries. (Notice the pool
>>>>>> with the small queries has the max number of running queries set
>>>>>> and the pool with the fewer but larger big queries has the max
>>>>>> memory set -- that is intentional, the former is faster for
>>>>>> admission but doesn't limit based on memory.) How well this will
>>>>>> work depends on how well you can pick good numbers for these
>>>>>> settings, which can be difficult and requires studying your workload.
>>>>>> This isn't perfect resource isolation because rogue queries can
>>>>>> still consume too much CPU or other resources, but it's the best
>>>>>> you'll be able to do right now. In the future we will have better
>>>>>> tools to make this easier.
>>>>>> Best,
>>>>>> Matt
>>>>>> On Mon, Jul 18, 2016 at 2:59 AM, 廖松博 <>
>>>>>>> Hello guys,
>>>>>>>        My Company is using Cloudera Impala as our basic
>>>>>>> infrastructure for online data analysis. The most difficult part
>>>>>>> we met is resource isolation and instability.
>>>>>>> According to our experiences in Impala, some big query which
>>>>>>> consume a vast amount of memory will crash impalad
>>>>>>> process(actually as worker but not coordinator, right?).
>>>>>>> In our simplest scenario, user A is a very important customer
>>>>>>> his queries are relatively small, user B is a unimportant user
>>>>>>> who may issue very large SQL to impala. It is unacceptable that
>>>>>>> the big query from user B crash the impalad process and affect
>>>>>>> the user experiences of user A. So resource isolation is the
>>>>>>> But per the Impala documents :
>>>>>>> a l a _ admission.html , Impala resource isolation is soft limit,
>>>>>>> cannot strictly prevent query from user B affecting user A.
>>>>>>> As I know llama(run impala with yarn) is not recommended and
>>>>>>> actually tried it but disappointed about the performance and
>>>>>>>        Is there any best practice for user resource isolation?
>>>>>>> different user will not affect each other.
>>>>>>>        Thanks.
>>>>>>> Best Regards,
>>>>>>> Songbo

View raw message