flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Yan <yanyan300...@gmail.com>
Subject Cannot download Jars from S3 due to resource timestamp changed
Date Thu, 04 Apr 2019 21:22:44 GMT

I am running issues when trying to move from HDFS to S3 using Flink 1.6.

I am getting an exception from Hadoop code:

IOException("Resource " + sCopy +
    " changed on src filesystem (expected " + resource.getTimestamp() +
    ", was " + sStat.getModificationTime());

Digging into this, I found there was one commit
by Nico trying to fix this issue in 2018. However, the fix did not work for
my case, as the fs.setTimes() method was not implemented in the hadoop-aws
S3AFilesystem I am using. And it seems S3 does not allow you to override
the last modified time for an object.

I am able to make an workaround the other way round: reading the timestamp
from S3 and override the local resource. Just wonder if any one has seen
similar issues, or he/she can actually make it work by using different
implementation of S3AFilesystem? Thanks!


View raw message