flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7989) flink-dist jar is deployed twice onto YARN
Date Mon, 06 Nov 2017 10:09:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240098#comment-16240098
] 

ASF GitHub Bot commented on FLINK-7989:
---------------------------------------

GitHub user NicoK opened a pull request:

    https://github.com/apache/flink/pull/4951

    [FLINK-7989][yarn] do not deploy the dist.jar file twice

    ## What is the purpose of the change
    
    We always add the dist.jar ourselves, but it could also be inside a shipped folder such
as the "lib/" folder and was then distributed multiple times.
    
    This PR is based on #4939 which changed the artefact uploads a bit.
    
    ## Brief change log
    
    - also exclude the `flink-dist*.jar` from any (recursive) upload of a shipped directory,
e.g. `lib`
    
    ## Verifying this change
    
    This change is already covered by existing tests, such as any test deploying a program
on YARN (if it didn't include the `flink-dist*.jar` file, it would fail). Additionally, a
manual deployment showed that `flink-dist*.jar` is only uploaded once.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (JavaDocs)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NicoK/flink flink-7989

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/4951.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4951
    
----
commit 6d40434b90d654df55dff3d267721743adc4f586
Author: Nico Kruber <nico@data-artisans.com>
Date:   2016-11-09T20:04:50Z

    [FLINK-4228][yarn/s3] fix for yarn staging with s3a defaultFs
    
    + includes a new unit tests for recursive uploads to hfds:// targets
    + add a unit test for recursive file uploads to s3:// via s3a

commit df9f9b855185bafe89ac09f4accbd603807e3f56
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-11-03T14:29:07Z

    [FLINK-4228][yarn/s3] turn the dependencies around
    
    Instead of having flink-s3-fs-hadoop depend on flink-yarn_<scala_version>,
    let flink-yarn depend on the s3 filesystem and implement the test there.
    This is safer with regards to the scala-independent flink-s3-fs-hadoop project.

commit e5488ef719573da0fe86b8355a8b997dd3e3d80a
Author: Nico Kruber <nico@data-artisans.com>
Date:   2017-11-03T20:19:34Z

    [FLINK-7989][yarn] do not add the dist.jar file twice
    
    We always add the dist.jar ourselves, but it could also be inside a shipped
    folder such as the "lib/" folder and was then distributed multiple times.

----


> flink-dist jar is deployed twice onto YARN
> ------------------------------------------
>
>                 Key: FLINK-7989
>                 URL: https://issues.apache.org/jira/browse/FLINK-7989
>             Project: Flink
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>
> We always add the dist.jar ourselves, but it could also be inside a shipped folder such
as the "lib/" folder and is then deployed again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message