flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6020) Blob Server cannot handle multiple job submits (with same content) parallelly
Date Mon, 08 May 2017 16:10:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001004#comment-16001004
] 

ASF GitHub Bot commented on FLINK-6020:
---------------------------------------

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/3525
  
    @WangTaoTheTonic I think we can solve that the following way:
    
      - The local upload uses `ATOMIC_MOVE` to rename the file
      - Only the thread that succeeds will store the blob in HDFS or S3
    
    What do you think?


> Blob Server cannot handle multiple job submits (with same content) parallelly
> -----------------------------------------------------------------------------
>
>                 Key: FLINK-6020
>                 URL: https://issues.apache.org/jira/browse/FLINK-6020
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Distributed Coordination
>            Reporter: Tao Wang
>            Assignee: Tao Wang
>            Priority: Critical
>
> In yarn-cluster mode, if we submit one same job multiple times parallelly, the task will
encounter class load problem and lease occuputation.
> Because blob server stores user jars in name with generated sha1sum of those, first writes
a temp file and move it to finalialize. For recovery it also will put them to HDFS with same
file name.
> In same time, when multiple clients sumit same job with same jar, the local jar files
in blob server and those file on hdfs will be handled in multiple threads(BlobServerConnection),
and impact each other.
> It's better to have a way to handle this, now two ideas comes up to my head:
> 1. lock the write operation, or
> 2. use some unique identifier as file name instead of ( or added up to) sha1sum of the
file contents.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message