flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Yan <yanyan300...@gmail.com>
Subject Cannot download Jars from S3 due to resource timestamp changed
Date Thu, 04 Apr 2019 21:22:44 GMT
Hi,

I am running issues when trying to move from HDFS to S3 using Flink 1.6.

I am getting an exception from Hadoop code:

IOException("Resource " + sCopy +
    " changed on src filesystem (expected " + resource.getTimestamp() +
    ", was " + sStat.getModificationTime());


Digging into this, I found there was one commit
<https://github.com/apache/flink/commit/c90a757b29f168144b1bae99df532911ae682e63>
made
by Nico trying to fix this issue in 2018. However, the fix did not work for
my case, as the fs.setTimes() method was not implemented in the hadoop-aws
S3AFilesystem I am using. And it seems S3 does not allow you to override
the last modified time for an object.

I am able to make an workaround the other way round: reading the timestamp
from S3 and override the local resource. Just wonder if any one has seen
similar issues, or he/she can actually make it work by using different
implementation of S3AFilesystem? Thanks!

-- 
Best,
Yan

Mime
View raw message