drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5737) Hash Agg uses more than the allocated memory under certain low memory conditions
Date Wed, 23 Aug 2017 18:44:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138866#comment-16138866

ASF GitHub Bot commented on DRILL-5737:

Github user sohami commented on the issue:

    @Ben-Zvi - Please help to review this PR.

> Hash Agg uses more than the allocated memory under certain low memory conditions
> --------------------------------------------------------------------------------
>                 Key: DRILL-5737
>                 URL: https://issues.apache.org/jira/browse/DRILL-5737
>             Project: Apache Drill
>          Issue Type: Bug
>            Reporter: Sorabh Hamirwasia
>            Assignee: Sorabh Hamirwasia
> Reported by [~rkins]
> Based on the memory computations drill thinks that there is not sufficient memory and
falls back to a single partition case. The single partition case however does not respect
the memory constraints imposed and completes the query using ~130MB of memory
> {code:java}
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 117127360;
> select count(*) from (select max(nulls_col), max(length(nulls_col)), max(`filename`)
from dfs.`/drill/testdata/hash-agg/data1` group by no_nulls_col) d;
> {code}
> Based on analysis by [~ben-zvi] this is by design. When the Hash Aggr Op finds that there
is not enough memory for at least two partitions, it falls back to the pre 1.11 behavior (
using 10GB limit ). 
> Solution is to provide a configuration based on which the fallback will be either allowed
or query will be failed.

This message was sent by Atlassian JIRA

View raw message