flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6916) FLIP-19: Improved BLOB storage architecture
Date Fri, 23 Jun 2017 22:07:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061567#comment-16061567

ASF GitHub Bot commented on FLINK-6916:

GitHub user NicoK opened a pull request:


    [FLINK-6916][blob] add API to allow job-related BLOBs to be stored     

    This PR is the fourth in a series for FLIP-19, based upon #4174. It allows (CONTENT_ADDRESSABLE)
BLOBs to not only be stored job-independently but also based on a jobId which will become
the future default for JARs, `TaskDeploymentDescriptor` data, and `DistributedCache` data.
    Please note that we will rework/reorganise the whole BLOB store APIs completely in a future
request. This PR focusses on the added functionality. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NicoK/flink flink-6916-3-jobid

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4176
commit ce719ee39fbbca7b7828c17d9792fc87d37450c7
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-01-06T17:42:58Z

    [FLINK-6008][docs] update some config options to the new, non-deprecated ones

commit 9efa8808e46adc1253f52a6a8cec6d3b4d29fee3
Author: Nico Kruber <nico@data-artisans.com>
Date:   2016-12-20T15:49:57Z

    [FLINK-6008][docs] minor improvements in the BlobService docs

commit ca3d533b0affa645ec93d40de378dadc829bbfe5
Author: Nico Kruber <nico@data-artisans.com>
Date:   2016-12-20T17:27:13Z

    [FLINK-6008] refactor BlobCache#getURL() for cleaner code

commit 0eededeb36dd833835753def7f4bb27c9d5fb67e
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-03-09T17:14:02Z

    [FLINK-6008] use Preconditions.checkArgument in BlobClient

commit 6249041a9db2b39ddf54e79a1aed5e7706e739c7
Author: Nico Kruber <nico@data-artisans.com>
Date:   2016-12-21T15:23:29Z

    [FLINK-6008] do not fail the BlobServer if delete fails
    also extend the delete tests and remove one code duplication

commit e681239a538547f752d65358db1ebd2ba312b33c
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-03-17T15:21:40Z

    [FLINK-6008] fix concurrent job directory creation
    also add according unit tests

commit 20beae2dbc91859e2ec724b35b20536dcd11fe90
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-04-18T14:37:37Z

    [FLINK-6008] some comments about BlobLibraryCacheManager cleanup

commit 8a33517fe6eb2fa932ab17cb0d82a3fa8d7b8d0b
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-04-19T13:39:03Z

    [hotfix] minor typos

commit 23889866ac21494fc4af90905ab1518cbe897118
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-04-19T14:10:16Z

    [FLINK-6008] further cleanup tests for BlobLibraryCacheManager

commit 01b1a245528c264a6061ed3a48b24c5a207369f6
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-14T16:01:47Z

    [FLINK-6008] do not guard a delete() call with a check for existence

commit cb249759b79d88eda37a8bb149040be3052059ac
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-16T08:51:04Z

    [FLINK-6916][blob] remove (unused) NAME_ADDRESSABLE mode

commit 7d3333c9cdb5bcf19a5e6b3190ce564201f1cc24
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T15:05:57Z

    [FLINK-6916][blob] remove further unused code due to the NAME_ADDRESSABLE removal

commit 00242371fed84a658ce88765204c450cc7819cf3
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-22T15:31:17Z

    [FLINK-6916] remove code duplication in BlobClientSslTest
    This lets BlobClientSslTest extend BlobClientTest as most of its implementation
    came from there and was simply copied.

commit 2a251e5cef0b757333d0dc1ff37f78fa0ea6eba7
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T12:45:31Z

    [FLINK-6916] remove LibraryCacheManager#getFile()
    This was only used in tests where it is avoidable but if used anywhere else, it
    may have caused cleanup issues.

commit 12d76e96247fc34a498260d323c67cbd07f3f905
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T14:14:15Z

    [FLINK-6916][blob] refactor getURL() to the more generic getFile()
    The fact that we always returned URL objects is a relic of the BlobServer's only
    use for URLClassLoader. Since we'd like to extend its use, returning File
    objects instead is more generic.

commit 3126c3c5e0c146e16d415fe75ccbfaf1770a3bea
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-23T09:40:34Z

    [FLINK-6916][blob] verify some of the buffers returned by GET

commit 4992ad7bb093444356d4258f2abe0731e1c18825
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-23T10:04:10Z

    [FLINK-6916][blob] use TemporaryFolder for local BLOB dir in unit tests
    This replaces the use of some temporary directory where it is not guaranteed
    that it will be deleted after the test.

commit d87d742b9a19efc1a26d2dda0c724d286ba9904d
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-21T16:04:43Z

    [FLINK-6916][blob] add API to allow job-related BLOBs to be stored

commit 053f4ba218e10038ddb4fa8790bb171144f70a72
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-06-23T17:17:07Z

    [FLINK-6916][blob] refactor the new API for job-related BLOBs
    For a cleaner API, instead of having a nullable jobId parameter, use two methods:
    one for job-related BLOBs, another for job-unrelated ones.


> FLIP-19: Improved BLOB storage architecture
> -------------------------------------------
>                 Key: FLINK-6916
>                 URL: https://issues.apache.org/jira/browse/FLINK-6916
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
> The current architecture around the BLOB server and cache components seems rather patched
up and has some issues regarding concurrency ([FLINK-6380]), cleanup, API inconsistencies
/ currently unused API ([FLINK-6329], [FLINK-6008]). These make future integration with FLIP-6
or extensions like offloading oversized RPC messages ([FLINK-6046]) difficult. We therefore
propose an improvement on the current architecture as described below which tackles these
issues, provides some cleanup, and enables further BLOB server use cases.
> Please refer to https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture
for a full overview on the proposed changes.

This message was sent by Atlassian JIRA

View raw message