hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "perry wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-6179) OOM occurs when query spans to a large number of partitions
Date Mon, 01 Sep 2014 05:57:21 GMT

     [ https://issues.apache.org/jira/browse/HIVE-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

perry wang updated HIVE-6179:
-----------------------------
    Description: 
When executing a query against a large number of partitions, such as "select count(*) from
table", OOM error may occur because Hive fetches the metadata for all partitions involved
and tries to store it in memory.
{code}
2014-01-09 13:14:17,090 ERROR metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(141))
- java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
        at java.lang.StringBuffer.append(StringBuffer.java:237)
        at org.apache.derby.impl.sql.conn.GenericStatementContext.appendErrorInfo(Unknown
Source)
        at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source) 
        at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:191)
        at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:379)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.loopJoinOrderedResult(MetaStoreDirectSql.java:641)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:410)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:205)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1433)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1420)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:122)
        at com.sun.proxy.$Proxy7.getPartitions(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:2128)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
{code}
The above error happened when executing "select count(*)" on a table with 40K partitions.

  was:
When executing a query against a large number of partitions, such as "select count(*) from
table", OOM error may occur because Hive fetches the metadata for all partitions involved
and tries to store it in memory.
{code}
2014-01-09 13:14:17,090 ERROR metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(141))
- java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2367)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
        at java.lang.StringBuffer.append(StringBuffer.java:237)
        at org.apache.derby.impl.sql.conn.GenericStatementContext.appendErrorInfo(Unknown
Source)
        at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
        at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:191)
        at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:379)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.loopJoinOrderedResult(MetaStoreDirectSql.java:641)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:410)
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:205)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1433)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1420)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:122)
        at com.sun.proxy.$Proxy7.getPartitions(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:2128)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
{code}
The above error happened when executing "select count(*)" on a table with 40K partitions.


> OOM occurs when query spans to a large number of partitions
> -----------------------------------------------------------
>
>                 Key: HIVE-6179
>                 URL: https://issues.apache.org/jira/browse/HIVE-6179
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.12.0
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>
> When executing a query against a large number of partitions, such as "select count(*)
from table", OOM error may occur because Hive fetches the metadata for all partitions involved
and tries to store it in memory.
> {code}
> 2014-01-09 13:14:17,090 ERROR metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(141))
- java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:2367)
>         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>         at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>         at java.lang.StringBuffer.append(StringBuffer.java:237)
>         at org.apache.derby.impl.sql.conn.GenericStatementContext.appendErrorInfo(Unknown
Source)
>         at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown
Source)
>         at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
>         at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
>         at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown
Source)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source) 
>         at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:191)
>         at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:379)
>         at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.loopJoinOrderedResult(MetaStoreDirectSql.java:641)
>         at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:410)
>         at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:205)
>         at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1433)
>         at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1420)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:122)
>         at com.sun.proxy.$Proxy7.getPartitions(Unknown Source)
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions(HiveMetaStore.java:2128)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
> {code}
> The above error happened when executing "select count(*)" on a table with 40K partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message