hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenning Ding (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HUDI-515) Resolve API conflict for Hive 2 and Hive 3
Date Fri, 31 Jan 2020 18:51:00 GMT

     [ https://issues.apache.org/jira/browse/HUDI-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenning Ding updated HUDI-515:
------------------------------
    Description: 
Currently I am working on supporting HIVE 3. There is an API issue.

In *HoodieCombineHiveInputFormat.java*, it calls a Hive method: HiveFileFormatUtils.getPartitionDescFromPathRecursively().
But this method is removed in Hive 3 and replaced by HiveFileFormatUtils.getFromPathRecursively().

Ideally, Hudi should support both Hive 2 & Hive 3 so that both
{code:java}
mvn clean install{code}
and
{code:java}
mvn clean install -Dhive.version=3.x{code}
could work.

One solution is to directly copy source code from [HiveFileFormatUtils.getFromPathRecursively()|https://github.com/apache/hive/blob/release-3.1.2-rc0/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java]
and put that code inside *HoodieCombineHiveInputFormat.java*.

The other way is use java reflection to decide which method to use. (preferred)

  was:
Currently I am working on supporting HIVE 3. There is an API issue.

In *HoodieCombineHiveInputFormat.java*, it calls a Hive method: HiveFileFormatUtils.getPartitionDescFromPathRecursively().
But this method is removed in Hive 3 and replaced by HiveFileFormatUtils.getFromPathRecursively().

Ideally, Hudi should support both Hive 2 & Hive 3 so that both
{code:java}
mvn clean install{code}
and
{code:java}
mvn clean install -Dhive.version=3.x{code}
could work.

One solution is to directly copy source code from [HiveFileFormatUtils.getFromPathRecursively()|https://github.com/apache/hive/blob/release-3.1.2-rc0/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java]
and put that code inside *HoodieCombineHiveInputFormat.java*.


> Resolve API conflict for Hive 2 and Hive 3
> ------------------------------------------
>
>                 Key: HUDI-515
>                 URL: https://issues.apache.org/jira/browse/HUDI-515
>             Project: Apache Hudi (incubating)
>          Issue Type: Sub-task
>            Reporter: Wenning Ding
>            Priority: Major
>
> Currently I am working on supporting HIVE 3. There is an API issue.
> In *HoodieCombineHiveInputFormat.java*, it calls a Hive method: HiveFileFormatUtils.getPartitionDescFromPathRecursively().
But this method is removed in Hive 3 and replaced by HiveFileFormatUtils.getFromPathRecursively().
> Ideally, Hudi should support both Hive 2 & Hive 3 so that both
> {code:java}
> mvn clean install{code}
> and
> {code:java}
> mvn clean install -Dhive.version=3.x{code}
> could work.
> One solution is to directly copy source code from [HiveFileFormatUtils.getFromPathRecursively()|https://github.com/apache/hive/blob/release-3.1.2-rc0/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java]
and put that code inside *HoodieCombineHiveInputFormat.java*.
> The other way is use java reflection to decide which method to use. (preferred)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message