hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
Date Sat, 10 Aug 2013 22:20:51 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736079#comment-13736079
] 

Hudson commented on HIVE-5011:
------------------------------

FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/])
HIVE-5011 Dynamic partitioning in HCatalog broken on external tables (Sushanth Sowmyan, reviewed
by Daniel Dai) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1512031)
* /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalDynamicPartitioned.java
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalHCatNonPartitioned.java
* /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalPartitioned.java

                
> Dynamic partitioning in HCatalog broken on external tables
> ----------------------------------------------------------
>
>                 Key: HIVE-5011
>                 URL: https://issues.apache.org/jira/browse/HIVE-5011
>             Project: Hive
>          Issue Type: Bug
>          Components: HCatalog
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>            Priority: Critical
>         Attachments: HIVE-5011.patch
>
>
> Dynamic partitioning with HCatalog has been broken as a result of HCATALOG-500 trying
to support user-set paths for external tables.
> The goal there was to be able to support other custom destinations apart from the normal
"hive-style" partitions. However, it is not currently possible for users to set paths for
dynamic ptn writes, since we don't support any way for users to specify "patterns"(like, say
"$\{rootdir\}/$v1.$v2/") into which writes happen, only "locations", and the values for dyn.
partitions are not known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the output committer,
which means that even if we supported patterned-writes instead of location-writes, we still
have to do some more deep diving into the output committer code to support it.
> Thus, my current proposal is that we honour writes to user-specified paths for external
tables *ONLY* for static partition writes - i.e., if we can determine that the write is a
dyn. ptn. write, we will ignore the user specification. (Note that this does not mean we ignore
the table's external location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message