drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Boaz Ben-Zvi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5616) Hash Agg Spill: OOM while reading irregular varchar data
Date Wed, 28 Jun 2017 22:56:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067407#comment-16067407
] 

Boaz Ben-Zvi commented on DRILL-5616:
-------------------------------------

More detail about the above failure:
{code}
Unable to allocate buffer of size 16777216 (rounded from 13107200) due to memory limit. Current
allocation: 150601728
{code}
Maybe our code has to take into account "rounding up" of each individual (buffer for) column,
not the total size (like above - 16MB instead of 12.2).

> Hash Agg Spill: OOM while reading irregular varchar data
> --------------------------------------------------------
>
>                 Key: DRILL-5616
>                 URL: https://issues.apache.org/jira/browse/DRILL-5616
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.11.0
>            Reporter: Boaz Ben-Zvi
>            Assignee: Boaz Ben-Zvi
>             Fix For: 1.11.0
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> An OOM while aggregating a table of two varchar columns where sizes vary significantly
(  about 8 bytes long in average, but 250 bytes max )
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 327127360;
> select count( * ) from (select max(`filename`) from dfs.`/drill/testdata/hash-agg/data2`
group by no_nulls_col, nulls_col) d;
> {code}
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.
> OOM at Second Phase. Partitions: 2. Estimated batch size: 12255232. Planned batches:
0. Rows spilled so far: 434127447 Memory limit: 163563680 so far allocated: 150601728.
> Fragment 1:0
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message