flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manjusha Vuyyuru <vmanjusha....@gmail.com>
Subject Re: Failed to fetch BLOB - IO Exception
Date Tue, 23 Oct 2018 08:27:13 GMT
Hello,

Checkpointing to hdfs.
*state.backend.fs.checkpointdir: hdfs://flink-hdfs:9000/flink-checkpoints*
*state.checkpoints.num-retained: 2*

Thanks,
Manjusha


On Tue, Oct 23, 2018 at 1:05 PM Dawid Wysakowicz <dwysakowicz@apache.org>
wrote:

> Hi Manjusha,
>
> I am not sure what is wrong, but Nico or Till (cc'ed) might be able to
> help you.
>
> Best,
>
> Dawid
> On 23/10/2018 06:58, Manjusha Vuyyuru wrote:
>
> Hello All,
>
> I have a  job which fails lets say after every 14 days with IO Exception,
> failed to fetch blob.
> I submitted the job using command line using java jar.Below is the
> exception I'm getting:
>
> java.io.IOException: Failed to fetch BLOB d23d168655dd51efe4764f9b22b85a18/p-446f7e0137fd66af062de7a56c55528171d380db-baf0b6bce698d586a3b0d30c6e487d16
from flink-job-mamager/10.20.1.85:38147 and store it under /tmp/blobStore-e3e34fec-22d9-4b3c-b542-0c1e5cdcf896/incoming/temp-00000022
> 	at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:191)
> 	at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:177)
> 	at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:205)
> 	at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:119)
> 	at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:878)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: GET operation failed: Server side error: /tmp/blobStore-5535a94c-5bdd-41f3-878d-8320e53ba7c5/incoming/temp-00182356
> 	at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:253)
> 	at org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:166)
> 	... 6 more
> Caused by: java.io.IOException: Server side error: /tmp/blobStore-5535a94c-5bdd-41f3-878d-8320e53ba7c5/incoming/temp-00182356
> 	at org.apache.flink.runtime.blob.BlobClient.receiveAndCheckGetResponse(BlobClient.java:306)
> 	at org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:247)
> 	... 7 more
> Caused by: java.nio.file.NoSuchFileException: /tmp/blobStore-5535a94c-5bdd-41f3-878d-8320e53ba7c5/incoming/temp-00182356
> 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> 	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> 	at java.nio.file.Files.move(Files.java:1395)
> 	at org.apache.flink.runtime.blob.BlobUtils.moveTempFileToStore(BlobUtils.java:452)
> 	at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:521)
> 	at org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> 	at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
>
> All the configurations of blob are default, i didn't change anything.
>
> Can someone help me to fix this issue.
>
> Thanks,
>
> Manjusha
>
>

Mime
View raw message