hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mousom Dhar Gupta (Commented) (JIRA)" <>
Subject [jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
Date Mon, 26 Mar 2012 18:40:28 GMT


Mousom Dhar Gupta commented on HIVE-2907:

For ObjectStore::dropTable am dropping pratitions in batches. And for DDLTask::dropTable am
getting all partition names and then getting partition metadata from those names in batches.

The batch size is determined by a config parameter (set to 1000 to start with).

> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>                 Key: HIVE-2907
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.9.0
>         Environment: General. Hive Metastore bug.
>            Reporter: Mousom Dhar Gupta
>            Priority: Minor
>             Fix For: 0.9.0
>         Attachments: HIVE-2907.1.patch.txt
>   Original Estimate: 10h
>  Remaining Estimate: 10h
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/

> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/ encounter out
of memory errors 
> when dropping tables with lots of partitions because they try to load the metadata for
every partition into memory.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message