hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sohan Jain" <sohanj...@fb.com>
Subject Re: Review Request: HIVE-2213: Optimize get_partition_names_ps()
Date Mon, 13 Jun 2011 21:11:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/878/
-----------------------------------------------------------

(Updated 2011-06-13 21:11:38.325243)


Review request for hive and Paul Yang.


Changes
-------

-Refactored similar functions
-Renamed getPartitionNamesPs() to listPartitionNamesPs()
-Modified get_partitions_ps() and get_partitions_ps_with_auth() for a similar optimization


Summary
-------

If a table has a large number of partitions, get_partition_names_ps() make take a long time
to execute, because we get all of the partition names from the database. This is not very
memory efficient, and the operation can be pushed down to the JDO layer without getting all
of the names first.


This addresses bug HIVE-2213.
    https://issues.apache.org/jira/browse/HIVE-2213


Diffs (updated)
-----

  trunk/common/src/java/org/apache/hadoop/hive/common/FileUtils.java 1135227 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1135227 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1135227 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1135227 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 1135227 
  trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1135227


Diff: https://reviews.apache.org/r/878/diff


Testing
-------

Passes previous test cases for get_partition_names_ps() in TestHiveMetaStore.


Thanks,

Sohan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message