hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dima machlin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-6448) Java heap space pasring a query with many union all
Date Mon, 17 Feb 2014 13:22:22 GMT

     [ https://issues.apache.org/jira/browse/HIVE-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

dima machlin updated HIVE-6448:
-------------------------------

    Description: 
too many union all statements in a single query require massive amount of memory to process.
Under default HiveServer Xmx (256) we can run ~50 union alls.
It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
The error is :

java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
        at java.lang.StringBuilder.append(StringBuilder.java:119)
        at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
        at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
        at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
        at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)

Query example :
select * 
from (
select count(*) from default.dual union all
select count(*) from default.dual union all
......
select count(*) from default.dual) z



  was:
too many union all statements in a single query require massive amount of memory to process.
Under default HiveServer Xmx (256) we can run ~50 union alls.
It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
The error is :

java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
        at java.lang.StringBuilder.append(StringBuilder.java:119)
        at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
        at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
        at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
        at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)



> Java heap space pasring a query with many union all
> ---------------------------------------------------
>
>                 Key: HIVE-6448
>                 URL: https://issues.apache.org/jira/browse/HIVE-6448
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, Query Processor
>    Affects Versions: 0.11.0
>         Environment:  
>            Reporter: dima machlin
>
> too many union all statements in a single query require massive amount of memory to process.
> Under default HiveServer Xmx (256) we can run ~50 union alls.
> It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
> The error is :
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:2882)
>         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>         at java.lang.StringBuilder.append(StringBuilder.java:119)
>         at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
>         at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
>         at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
>         at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>         at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Query example :
> select * 
> from (
> select count(*) from default.dual union all
> select count(*) from default.dual union all
> ......
> select count(*) from default.dual) z



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message