flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5908) Blob Cache can (rarely) get corrupted on failed blob downloads
Date Fri, 24 Feb 2017 16:37:44 GMT
Stephan Ewen created FLINK-5908:

             Summary: Blob Cache can (rarely) get corrupted on failed blob downloads
                 Key: FLINK-5908
                 URL: https://issues.apache.org/jira/browse/FLINK-5908
             Project: Flink
          Issue Type: Bug
          Components: Distributed Coordination
    Affects Versions: 1.2.0
            Reporter: Stephan Ewen
             Fix For: 1.3.0, 1.2.1

The Blob Cache downloads files directly to the target file location.
While it tries to clean up failed attempts, there is a change that this cleanup does not complete.

In that case, we have a corrupt file at the target location. The blob cache then assumes that
it already has the file cached already and future requests do not attempt to re-download the

The fix would be to download to a temp file name, validate the integrity, and rename to the
target file path when the validation succeeds.

The validation for "content addressable" could even include validating the checksum hash.

This message was sent by Atlassian JIRA

View raw message