hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sohan Jain" <>
Subject Re: Review Request: HIVE-2246: Dedupe tables' column schemas from partitions in the metastore db
Date Mon, 08 Aug 2011 21:19:07 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated 2011-08-08 21:19:06.999293)

Review request for hive, Ning Zhang and Paul Yang.


revised description for latest changes

Summary (updated)

This patch tries to make minimal changes to the API while keeping migration short and somewhat
easy to revert.

The new schema can be described as follows:
- CDS is a table corresponding to Column Descriptor objects.  Currently, it only stores a
- COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns.  A Column Descriptor
holds a list of columns.  COLUMNS_V2 has a foreign key to the CD_ID to which it belongs.
- SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID
which describes its columns.

During migration, we create Column Descriptors for tables in a straightforward manner: their
columns are now just wrapped inside a column descriptor.  The SDS of partitions use their
parent table's column descriptor, since currently a partition and its table share the same
list of columns.

When altering or adding a partition, give it it's parent table's column descriptor IF the
columns they describe are the same.  Otherwise, create a new column descriptor for its columns.

When creating a table, create a new column descriptor every time.  When altering a table,
only construct a new column descriptor if the columns list has changed.

Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to
see if the related column descriptor has any other references in the table.  That is, check
to see if any other storage descriptors point to that column descriptor.  If none do, then
delete that column descriptor.  This check is in place so we don't have unreferenced column
descriptors and columns hanging around after schema evolution for tables.

This addresses bug HIVE-2246.


  trunk/metastore/scripts/upgrade/derby/008-HIVE-2246.derby.sql PRE-CREATION 
  trunk/metastore/scripts/upgrade/derby/008-REVERT-HIVE-2246.derby.sql PRE-CREATION 
  trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ 1153927 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ 1153927 
  trunk/metastore/src/model/package.jdo 1153927 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ 1153927 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/ 1153927 



Passes facebook's regression testing and all existing test cases.  In one instance, before
migration, the overhead involved with storage descriptors and columns was ~11 GB.  After migration,
the overhead was ~1.5 GB.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message