drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boaz Ben-Zvi <bben-...@mapr.com>
Subject Re: direct.used on the Metrics page exceeded planner.memory.max_query_memory_per_node while running a single query
Date Mon, 14 Aug 2017 19:00:22 GMT
Did your query include a hash join ?

As of 1.11, only the External Sort and Hash Aggregate operators obey the memory limit (that
is, the “max query memory per node” figure is divided among all the instances of these
operators).
The Hash Join (as was before 1.11) still does not take part in this memory allocation scheme,
and each instance may use up to 10GB.

Also in 1.11, the Hash Aggregate may “fall back” to the 1.10 behavior (same as the Hash
Join; i.e. up to 10GB) in case there is too little memory per an instance (because it cannot
perform memory spilling, which requires some minimal memory to hold multiple batches).

   Thanks,

          Boaz 

On 8/11/17, 4:25 PM, "Muhammad Gelbana" <m.gelbana@gmail.com> wrote:

    Sorry for the long subject !
    
    I'm running a single query on a single node Drill setup.
    
    I assumed that setting the *planner.memory.max_query_memory_per_node* property
    controls the max amount of memory (in bytes) for each running on a single
    node. Which means that in my setup, the *direct.used* metric in the metrics
    page should never exceed that value in my case.
    
    But it did and drastically. I assigned *34359738368* (32 GB) to the
    *planner.memory.max_query_memory_per_node* option but while monitoring the
    *direct.used* metric, I found that it reached *51640484458* (~48 GB).
    
    What did I mistakenly do\interpret ?
    
    Thanks,
    Gelbana
    ​​
    

Mime
View raw message