hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liujianhui (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HUDI-525) inserts info miss in delta_commit_inflight meta file
Date Mon, 13 Jan 2020 07:51:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014094#comment-17014094
] 

liujianhui edited comment on HUDI-525 at 1/13/20 7:50 AM:
----------------------------------------------------------

{code}

cat 20200113145851.deltacommit.inflight

{ "partitionToWriteStats" : \\{ }

,
 "compacted" : false,
 "extraMetadataMap" : \{ },
 "totalScanTime" : 0,
 "totalCreateTime" : 0,
 "totalUpsertTime" : 0,
 "totalCompactedRecordsUpdated" : 0,
 "totalLogFilesSize" : 0,
 "totalLogFilesCompacted" : 0,
 "fileIdAndRelativePaths" : \{ },
 "totalRecordsDeleted" : 0,
 "totalLogRecordsCompacted" : 0,
 "extraMetadata" : \{ }
 }

{code}

the deltacommit should contains the insert info, event the field id is unknown at that moment


was (Author: liujianhuiouc):
{ code}

cat 20200113145851.deltacommit.inflight
{
 "partitionToWriteStats" : \{ },
 "compacted" : false,
 "extraMetadataMap" : \{ },
 "totalScanTime" : 0,
 "totalCreateTime" : 0,
 "totalUpsertTime" : 0,
 "totalCompactedRecordsUpdated" : 0,
 "totalLogFilesSize" : 0,
 "totalLogFilesCompacted" : 0,
 "fileIdAndRelativePaths" : \{ },
 "totalRecordsDeleted" : 0,
 "totalLogRecordsCompacted" : 0,
 "extraMetadata" : \{ }
}

{code}

the deltacommit should contains the insert info, event the field id is unknown at that moment

> inserts info miss in delta_commit_inflight meta file
> ----------------------------------------------------
>
>                 Key: HUDI-525
>                 URL: https://issues.apache.org/jira/browse/HUDI-525
>             Project: Apache Hudi (incubating)
>          Issue Type: Bug
>            Reporter: liujianhui
>            Priority: Major
>
> should add  insert info in WorkInfoStat
> {code}
> private void saveWorkloadProfileMetadataToInflight(WorkloadProfile profile, HoodieTable<T>
table, String commitTime)
>  throws HoodieCommitException {
>  try {
>  HoodieCommitMetadata metadata = new HoodieCommitMetadata();
>  profile.getPartitionPaths().forEach(path -> {
>  WorkloadStat partitionStat = profile.getWorkloadStat(path.toString());
>  partitionStat.getUpdateLocationToCount().forEach((key, value) -> {
>  HoodieWriteStat writeStat = new HoodieWriteStat();
>  writeStat.setFileId(key);
>  // TODO : Write baseCommitTime is possible here ?
>  writeStat.setPrevCommit(value.getKey());
>  writeStat.setNumUpdateWrites(value.getValue());
>  metadata.addWriteStat(path.toString(), writeStat);
>  });
>  });
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message