hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <>
Subject [jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
Date Sun, 11 Aug 2013 00:01:51 GMT


Hudson commented on HIVE-5011:

SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [])
HIVE-5011 Dynamic partitioning in HCatalog broken on external tables (Sushanth Sowmyan, reviewed
by Daniel Dai) (khorgath:
* /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/

> Dynamic partitioning in HCatalog broken on external tables
> ----------------------------------------------------------
>                 Key: HIVE-5011
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>            Priority: Critical
>         Attachments: HIVE-5011.patch
> Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500 trying
to support user-set paths for external tables.
> The goal there was to be able to support other custom destinations apart from the normal
"hive-style" partitions. However, it is not currently possible for users to set paths for
dynamic ptn writes, since we don't support any way for users to specify "patterns"(like, say
"$\{rootdir\}/$v1.$v2/") into which writes happen, only "locations", and the values for dyn.
partitions are not known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the output committer,
which means that even if we supported patterned-writes instead of location-writes, we still
have to do some more deep diving into the output committer code to support it.
> Thus, my current proposal is that we honour writes to user-specified paths for external
tables *ONLY* for static partition writes - i.e., if we can determine that the write is a
dyn. ptn. write, we will ignore the user specification. (Note that this does not mean we ignore
the table's external location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message