flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7196) add a TTL to transient BLOB files
Date Fri, 21 Jul 2017 13:46:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096272#comment-16096272

ASF GitHub Bot commented on FLINK-7196:

GitHub user NicoK opened a pull request:


    [FLINK-7196][blob] add a TTL to all transient BLOBs

    ## What is the purpose of the change
    Transient BLOB files are currently only deleted manually and may thus linger around if
not cleaned up. We should introduce a default cleanup time, i.e. a time-to-live (TTL), as
a backup cleanup path.
    ## Brief change log
      - Any `TransientBlobService`, i.e. `BlobServer` and `TransientBlobCache`, gets an additional
member keeping track of all expiry times of transient BLOBs.
      - Whenever a transient BLOB enters the local store of a `TransientBlobService`, its
TTL will be set to `currentTime + cleanupInterval`.
      - A cleanup task (`TransientBlobCleanupTask`) is executed every `cleanupInterval` seconds
and deletes all expired transient BLOBs.
      - As a result, transient BLOBs stay at most `2 * cleanupInterval` seconds before getting
deleted automatically.
    Please note that this is based on #4359 and should conclude FLIP-19.
    ## Verifying this change
    This change adds tests and can be verified as follows:
      - a new test verifies (for each `BlobServer` and `TransientBlobCache` with and without
a job ID) that after adding two transient BLOBs with further accesses only to one of them,
that the accessed one remains and the other is deleted. For the existence check, however,
there is a slight race condition: if the test is slow and delayed by 1s, the cleanup may kick
in and the test may fail. This is hard to test otherwise without further intrusion into the
blob store classes.
      - FYI: since transient BLOBs are not used yet, this cannot be tested with a complete
flink distribution yet.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): **(no)**
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **(no)**
      - The serializers: **(no)**
      - The runtime per-record code paths (performance sensitive): **(no)**
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: **(yes)**:
    ## Documentation
      - Does this pull request introduce a new feature? **(yes)**
      - If yes, how is the feature documented? **(JavaDocs)**

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NicoK/flink flink-6916-7196

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4381
commit 1f86efd013a81e84ba1556de0a04e4ac70229f79
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-22T15:31:17Z

    [FLINK-7053][blob] remove code duplication in BlobClientSslTest
    This lets BlobClientSslTest extend BlobClientTest as most of its implementation
    came from there and was simply copied.

commit ff083e850edb8f5f383f54f19519116e10308d61
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-23T09:40:34Z

    [FLINK-7053][blob] verify some of the buffers returned by GET

commit e138569019f5c75b70a21085e4829cb6cd1e93bc
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-23T10:04:10Z

    [FLINK-7053][blob] use TemporaryFolder for local BLOB dir in unit tests
    This replaces the use of some temporary directory where it is not guaranteed
    that it will be deleted after the test.

commit 70f06d9b38c5d8afb50b6c18b7b3a4e07e2bb3da
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T12:45:31Z

    [FLINK-7054][blob] remove LibraryCacheManager#getFile()
    This was only used in tests where it is avoidable but if used anywhere else, it
    may have caused cleanup issues.

commit 5463f65aeb3ed519b51ccc78abfcc003bf02b3f8
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-16T21:13:56Z

    [FLINK-7054][hotfix] fix a checkstyle error

commit 12052c0b8c8b869f9b6bd6f9d53c9ed6e362c631
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T14:14:15Z

    [FLINK-7055][blob] refactor getURL() to the more generic getFile()
    The fact that we always returned URL objects is a relic of the BlobServer's only
    use for URLClassLoader. Since we'd like to extend its use, returning File
    objects instead is more generic.

commit 9dd7ba652ff1b09fb5bf44e8b6b5b885a56873ab
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T16:04:43Z

    [FLINK-7056][blob] add API to allow job-related BLOBs to be stored

commit 90f779ba66728979977bd794e8f68b207471dbc2
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-23T17:17:07Z

    [FLINK-7056][blob] refactor the new API for job-related BLOBs
    For a cleaner API, instead of having a nullable jobId parameter, use two methods:
    one for job-related BLOBs, another for job-unrelated ones.

commit 6d1fda80c6f9869676030fd86177bbc7e981b1f9
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-16T21:22:12Z

    [FLINK-7056][hotfix] fix a checkstyle error

commit cdbb0fb32052c8163bc83df151a0c8354846e4dc
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-27T16:29:44Z

    [FLINK-7057][blob] move ref-counting from the LibraryCacheManager to the BlobCache
    Also change from BlobKey-based ref-counting to job-based ref-counting which is
    simpler and the mode we want to use from now on. Deferred cleanup (as before)
    is currently not implemented yet (TODO).
    At the BlobServer, no ref-counting will be used but the cleanup will happen
    when the job enters a final state (TODO).

commit 8617c9e8ffbdece8d02815ef71dc09f2d764453a
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-28T09:31:39Z

    [FLINK-7057][blob] change to a cleaner API for BlobService#registerJob()

commit 20cf9bf1fd2b382b6d45dc7b15c7be9bed7f5f16
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-28T12:09:11Z

    [FLINK-7057][blob] implement deferred cleanup at the BlobCache
    Whenever a job is not referenced at the BlobCache anymore, we set a TTL and let
    the cleanup task remove it when this is hit and the task is run. For now, this
    means that a BLOB will be retained at most
    (2 * ConfigConstants.LIBRARY_CACHE_MANAGER_CLEANUP_INTERVAL) seconds after not
    being referenced anymore. We do this so that a recovery still has the chance to
    use existing files rather than to download them again.

commit 270f3957b6d5e9061d528158a86eb1837ad2786c
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-28T15:17:06Z

    [FLINK-7057][blob] integrate cleanup of job-related JARs from the BlobServer
    TODO: an integration test that verifies that this is actually done when desired
    and not performed when not, e.g. if the job did not reach a final execution

commit 39300a32c90c82f8044cfd6d5fcaf1b83b250c87
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-30T12:52:19Z

    [FLINK-7057][tests] extract FailingBlockingInvokable from CoordinatorShutdownTest

commit 56113349799669326074d269d6c07bb19bd45788
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-30T12:56:14Z

    [FLINK-7057][blob] add an integration test for the BlobServer cleanup
    This ensures that BLOB files are actually deleted when a job enters a final

commit 93861f1f624c5b514fcdd041b592e0c40c66d103
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-03T09:25:29Z

    [FLINK-7057][tests] refrain from catching an exception just to fail the test
    removes code like this in the BLOB store unit tests:
    catch (Exception e) {

commit 80a05142a68f8a3d0aaf1d447ab87a1e506364a4
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-03T11:45:33Z

    [FLINK-7057][blob] fix BlobServer#cleanupJob() being too eager
    Instead of deleting the job's directory, it was deleting the parent storage

commit de4ca2a7c676e7e1a82172ce680ae5a5f6693df4
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-03T15:03:50Z

    [FLINK-7057][blob] fix BlobServer cleanup integration
    * the test did not check the correct directories for cleanup
    * the test did not honour the test timeout

commit dc58fa5d5ffc299b493dc1a78f26aa1914036151
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-03T15:11:18Z

    [FLINK-7057][blob] test and fix BlobServer cleanup for a failed job submission

commit d1de87bfd34c5bb1f2dde0402bcceac6013c150b
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-04T07:55:29Z

    [FLINK-7057][blob] rework the LibraryCacheManager API
    Since ref-counting has moved to the BlobCache, the BlobLibraryCacheManager is
    just a thin wrapper to get a user class loader by retrieving BLOBs from the
    BlobCache/BlobServer. Therefore, move the job-registration/-release out of it,
    too, and restrict its use to the task manager where the BlobCache is used (on
    the BlobServer, jobs do not need registration since they are only used once and
    will be deleted when they enter a final state).
    This makes the BlobServer and BlobCache instances available at the JobManager
    and TaskManager instances, respectively, also enabling future use cases outside
    of the LibraryCacheManager.

commit b62f1f640e33126523eaaa7bd51f0b24962c67fd
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-04T09:50:07Z

    [hotfix] increase Scala checkstyle maxParameters to 20

commit 5e9a9a7cd779a8e0d28947bcfa64cc16c02857f2
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-04T09:51:28Z

    [FLINK-7057][blob] address PR comments

commit 5fa960676281fc1473105c849fbce9cb22cacacb
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-04T12:41:18Z

    [FLINK-7057][blob] fix JobManagerLeaderElectionTest

commit a6ca6812af7b4130e77775fd3896f3f44a357334
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-04T20:49:12Z

    [FLINK-7057][blob] re-introduce some ref-counting for BlobLibraryCacheManager
    Apparently, we do need to return the same ClassLoader for different (parallel)
    tasks of a job running on the same task manager. Therefore, keep the initial
    task registration implementation that was removed with
    8331fbb208d975e0c1ec990344c14315ea08dd4a and only adapt it here. This also
    restores some tests and adds new combinations not tested before.

commit da1be581529813ec359f1ac09a00bc564a3812dd
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-06T07:38:17Z

    [FLINK-7057][blob] address PR comments

commit b02807e75cea960fd2c89e49656be2e6f6e32394
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-10T11:40:00Z

    [FLINK-7057][tests] fix (manual/ignored) BlobCacheCleanupTest#testJobDeferredCleanup()

commit 0f9f7d7f90497708cd2589b34573f10be0b9d331
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-16T21:30:43Z

    [FLINK-7057][hotfix] fix a checkstyle error

commit ba540ae1445a8144aff389c5056f34b54a7cbbdb
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-20T09:16:38Z

    [FLINK-7057][blob] remove the extra lock object from BlobCache
    We can lock on jobRefCounters instead, which is what we are guarding anyway.

commit 871a590b5c1b3897e26e516e027a86345e3a3f4f
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-20T09:17:21Z

    [FLINK-7057][blob] minor improvements to the TTL in BlobCache
    Do not use Long.MAX_VALUE as a code for "keep forever". Also add more comments.

commit c48575fe81367810f2fd5ae082170f349d3e7176
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-07-20T15:12:24Z

    [FLINK-7057][blob] replace "library-cache-manager.cleanup.interval" with "blob.service.cleanup.interval"
    Since we moved the cleanup to the BLOB service classes, this only makes sense.


> add a TTL to transient BLOB files
> ---------------------------------
>                 Key: FLINK-7196
>                 URL: https://issues.apache.org/jira/browse/FLINK-7196
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Distributed Coordination, Network
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
> Transient BLOB files are not automatically cleaned up unless the {{BlobCache}}/{{BlobServer}}
are shut down or the files are deleted via the {{delete}} methods. Additionally, they should
have a default time-to-live (TTL) so that they may be cleaned up in failure cases.

This message was sent by Atlassian JIRA

View raw message