hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] bwu2 edited a comment on issue #1128: [HUDI-453] Fix throw failed to archive commits error when writing data to MOR/COW table
Date Wed, 04 Mar 2020 22:53:42 GMT
bwu2 edited a comment on issue #1128: [HUDI-453] Fix throw failed to archive commits error
when writing data to MOR/COW table
URL: https://github.com/apache/incubator-hudi/pull/1128#issuecomment-594824906
 
 
   It's not easily reproducible. It seems to be a sporadic thing that happens from time to
time; from the 95 tables we ingest (every 4 hours) for the last couple of weeks, the error
has occurred 4 or 5 times. The jobs that created these zero bytes files seem to complete successfully
and without any error.
   
   In the Spark logs for the job, things seem ok (this is the only mention of `.clean` files):
   `20/02/21 21:11:13 INFO S3NativeFileSystem2: rename s3://bucket/path/to/table/.hoodie/20200221211054.clean.inflight
s3://bucket/path/to/table/.hoodie/20200221211054.clean`
   
   The s3 `.hoodie` folder looks like (ignore the dates on the files, i copied and pasted
from the original folder; also, the listing is truncated): 
   ![image](https://user-images.githubusercontent.com/14202231/75920973-2ec1d300-5e15-11ea-9794-ab297fdf9b9f.png)
   and the archived folder looks like:
   ![image](https://user-images.githubusercontent.com/14202231/75921034-49944780-5e15-11ea-9c86-5d090c668fcc.png)
   
   We didn't actually notice the error until much later when the cleaning fails because of
some old zero byte files.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message