hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pratyaksh Sharma (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HUDI-485) Check for where clause is wrong in HiveIncrementalPuller
Date Thu, 02 Jan 2020 05:51:00 GMT

     [ https://issues.apache.org/jira/browse/HUDI-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pratyaksh Sharma updated HUDI-485:
----------------------------------
    Description: 
HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 

if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'"))

{ LOG.info("Incremental SQL : " + incrementalSQL + " does not contain `_hoodie_commit_time`
> %targetBasePath. Please add " + "this clause for incremental to work properly."); throw
new HoodieIncrementalPullSQLException( "Incremental SQL does not have clause `_hoodie_commit_time`
> '%targetBasePath', which " + "means its not pulling incrementally"); }

Basically we are trying to add a placeholder here which is later replaced with config.fromCommitTime
here - 

incrementalPullSQLtemplate.add("incrementalSQL", String.format(incrementalSQL, config.fromCommitTime));

Hence, the above check needs to replaced with `_hoodie_commit_time` > %s

  was:
HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 

if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'")) {
 LOG.info("Incremental SQL : " + incrementalSQL
 + " does not contain `_hoodie_commit_time` > %targetBasePath. Please add "
 + "this clause for incremental to work properly.");
 throw new HoodieIncrementalPullSQLException(
 "Incremental SQL does not have clause `_hoodie_commit_time` > '%targetBasePath', which
"
 + "means its not pulling incrementally");
}

Basically we are trying to add a placeholder here which is later replaced with config.fromCommitTime
here - 

incrementalPullSQLtemplate.add("incrementalSQL", String.format(incrementalSQL, config.fromCommitTime));

Hence, the above check needs to replaced with `_hoodie_commit_time` > %targetBasePath


> Check for where clause is wrong in HiveIncrementalPuller
> --------------------------------------------------------
>
>                 Key: HUDI-485
>                 URL: https://issues.apache.org/jira/browse/HUDI-485
>             Project: Apache Hudi (incubating)
>          Issue Type: Sub-task
>          Components: Incremental Pull, newbie
>            Reporter: Pratyaksh Sharma
>            Assignee: Pratyaksh Sharma
>            Priority: Major
>
> HiveIncrementalPuller checks the clause in incrementalSqlFile like this -> 
> if (!incrementalSQL.contains("`_hoodie_commit_time` > '%targetBasePath'"))
> { LOG.info("Incremental SQL : " + incrementalSQL + " does not contain `_hoodie_commit_time`
> %targetBasePath. Please add " + "this clause for incremental to work properly."); throw
new HoodieIncrementalPullSQLException( "Incremental SQL does not have clause `_hoodie_commit_time`
> '%targetBasePath', which " + "means its not pulling incrementally"); }
> Basically we are trying to add a placeholder here which is later replaced with config.fromCommitTime
here - 
> incrementalPullSQLtemplate.add("incrementalSQL", String.format(incrementalSQL, config.fromCommitTime));
> Hence, the above check needs to replaced with `_hoodie_commit_time` > %s



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message