drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pritesh Maker (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (DRILL-5891) When Drill runs out of memory for a HashAgg, it should tell the user how much memory to allocate
Date Tue, 24 Oct 2017 12:58:00 GMT

     [ https://issues.apache.org/jira/browse/DRILL-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pritesh Maker reassigned DRILL-5891:
------------------------------------

    Assignee: Boaz Ben-Zvi  (was: Pritesh Maker)

> When Drill runs out of memory for a HashAgg, it should tell the user how much memory
to allocate
> ------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-5891
>                 URL: https://issues.apache.org/jira/browse/DRILL-5891
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.11.0
>            Reporter: Robert Hou
>            Assignee: Boaz Ben-Zvi
>
> Query is:
> select count(\*), max(`filename`) from dfs.`/drill/testdata/hash-agg/data1` group by
no_nulls_col, nulls_col;
> Error is:
> Error: RESOURCE ERROR: Not enough memory for internal partitioning and fallback mechanism
for HashAgg to use unbounded memory is disabled. Either enable fallback config drill.exec.hashagg.fallback.enabled
using Alter session/system command or increase memory limit for Drillbit
> From drillbit.log:
> {noformat}
> 2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE o.a.d.e.p.i.aggregate.HashAggregator
- Incoming sizer: Actual batch schema & sizes {
>   no_nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size: 130, data
size: 132892)
>   nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size: 112, data
size: 113673)
>   EXPR$0(type: REQUIRED BIGINT, count: 1023, std size: 8, actual size: 8, data size:
8184)
>   EXPR$1(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size: 18, data size:
18414)
>   Records: 1023, Total size: 524288, Data size: 273163, Gross row width: 513, Net row
width: 268, Density: 53%}
> 2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE o.a.d.e.p.i.aggregate.HashAggregator
- 2nd phase. Estimated internal row width: 166 Values row width: 66 batch size: 12779520 
memory limit: 63161283  max column width: 50
> 2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:3:2] TRACE o.a.d.e.p.impl.common.HashTable
- HT allocated 4784128 for varchar of max width 50
> 2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:15] INFO  o.a.d.e.p.i.aggregate.HashAggregator
- User Error Occurred: Not enough memory for internal partitioning and fallback mechanism
for HashAgg to use unbounded memory is disabled. Either enable fallback config drill.exec.hashagg.fallback.enabled
using Alter session/system command or increase memory limit for Drillbit
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Not enough memory for
internal partitioning and fallback mechanism for HashAgg to use unbounded memory is disabled.
Either enable fallback config drill.exec.hashagg.fallback.enabled using Alter session/system
command or increase memory limit for Drillbit
> {noformat}
> I would recommend that we add a log message with the "alter" command to increase the
amount of memory allocated, and how much memory to allocate.  Otherwise, the user may not
know what to do.
> I would also not suggest enabling "drill.exec.hashagg.fallback.enabled" except as a last
resort.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message