hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <>
Subject [jira] [Created] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
Date Tue, 06 Aug 2013 20:07:47 GMT
Sushanth Sowmyan created HIVE-5011:

             Summary: Dynamic partitioning in HCatalog broken on external tables
                 Key: HIVE-5011
             Project: Hive
          Issue Type: Bug
          Components: HCatalog
            Reporter: Sushanth Sowmyan
            Assignee: Sushanth Sowmyan

Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500 trying to support
user-set paths for external tables.

The goal there was to be able to support other custom destinations apart from the normal "hive-style"
partitions. However, it is not currently possible for users to set paths for dynamic ptn writes,
since we don't support any way for users to specify "patterns"(like, say "${rootdir}/$v1.$v2/")
into which writes happen, only "locations", and the values for dyn. partitions are not known
ahead of time. Also, specifying a custom path messes with the way dynamic ptn. code tries
to determine what was written to where from the output committer, which means that even if
we supported patterned-writes instead of location-writes, we still have to do some more deep
diving into the output committer code to support it.

Thus, my current proposal is that we honour writes to user-specified paths for external tables
*ONLY* for static partition writes - i.e., if we can determine that the write is a dyn. ptn.
write, we will ignore the user specification. (Note that this does not mean we ignore the
table's external location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message