hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balaji Varadarajan (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HUDI-716) Exception: Not an Avro data file when running HoodieCleanClient.runClean
Date Wed, 18 Mar 2020 18:26:00 GMT

    [ https://issues.apache.org/jira/browse/HUDI-716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061977#comment-17061977
] 

Balaji Varadarajan commented on HUDI-716:
-----------------------------------------

Spoke with Alex to debug this issue. It has to do with failed clean operation before upgrade
to 0.5.[1,2].  Here is the context

```

Before 0.5.1, hudi was handling clean differently than commit action. If a clean failed during
final step, there would be .clean.inflight files (empty or corrupted) lying around but pre
0.5.1 did not care. With 0.5.1 onwards, we are handling clean action consistently like commit.
Hudi would first scan all files and store the plan (atomically) in .clean.requested and .clean.inflight
before triggering the actual clean. If there are any intermittent failures, subsequent clean
would read the cleaner plan again and retry.

 

In this case, Cleaner is reading an empty .clean.inflight plan and looking for Cleaner plan
but it is empty. 

```

One workaround would be to catch exception in the below section and delete pending clean

```

// If there are inflight(failed) or previously requested clean operation, first perform them
table.getCleanTimeline().filterInflightsAndRequested().getInstants().forEach(hoodieInstant
-> {
 LOG.info("There were previously unfinished cleaner operations. Finishing Instant=" + hoodieInstant);
 runClean(table, hoodieInstant);
});

```

 

 

> Exception: Not an Avro data file when running HoodieCleanClient.runClean
> ------------------------------------------------------------------------
>
>                 Key: HUDI-716
>                 URL: https://issues.apache.org/jira/browse/HUDI-716
>             Project: Apache Hudi (incubating)
>          Issue Type: Bug
>          Components: DeltaStreamer
>            Reporter: Alexander Filipchik
>            Assignee: lamber-ken
>            Priority: Major
>             Fix For: 0.6.0
>
>
> Just upgraded to upstream master from 0.5 and seeing an issue at the end of the delta
sync run: 
> 20/03/17 02:13:49 ERROR HoodieDeltaStreamer: Got error running delta sync once. Shutting
down20/03/17 02:13:49 ERROR HoodieDeltaStreamer: Got error running delta sync once. Shutting
downorg.apache.hudi.exception.HoodieIOException: Not an Avro data file at org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:144)
at org.apache.hudi.client.HoodieCleanClient.lambda$clean$0(HoodieCleanClient.java:88) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at org.apache.hudi.client.HoodieCleanClient.clean(HoodieCleanClient.java:86)
at org.apache.hudi.client.HoodieWriteClient.clean(HoodieWriteClient.java:843) at org.apache.hudi.client.HoodieWriteClient.postCommit(HoodieWriteClient.java:520)
at org.apache.hudi.client.AbstractHoodieWriteClient.commit(AbstractHoodieWriteClient.java:168)
at org.apache.hudi.client.AbstractHoodieWriteClient.commit(AbstractHoodieWriteClient.java:111)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:395) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:237)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:121)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused
by: java.io.IOException: Not an Avro data file at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:50)
at org.apache.hudi.common.util.AvroUtils.deserializeAvroMetadata(AvroUtils.java:147) at org.apache.hudi.common.util.CleanerUtils.getCleanerPlan(CleanerUtils.java:87)
at org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:141) ... 24 more
>  
> It is attempting to read an old cleanup file (2 month old) and crashing
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message