flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
Date Tue, 06 Oct 2015 11:10:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944882#comment-14944882
] 

ASF GitHub Bot commented on FLINK-2797:
---------------------------------------

Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145824055
  
    Yes. I agree. It started out as a simple issue to support detached jobs but gotten a bit
messy.
    1. In detached mode, whether the program is run in interactive mode or normal mode, if
there is a call to an eager execution function, it will lead to an exception. [As of now.]
    2. This is where the problem started to appear more difficult. Since the accumulator addition
is entirely handled by Runtime Context, there can be no check whether the job is in detached
mode. Of course, having to store this information in two places[`ExecutionEnvironment` and
`RuntimeContext`] is poor design. 
    If the job is being run in detached mode, the accumulators are lost and user can never
access them or use them.
    But disallowing accumulators will effectively break most of existing programs. So that's
bad choice.
    Eager execution functions use just accumulators actually, so, to maintain harmony, we
have to maintain them too.
    If we just consider the first commit of this PR which adds support for executing detached
jobs, on a eager call, the main program throws an exception, because the job execution result
doesn't contain any accumulators.
    ```java
    this.flatMap(new Utils.CollectHelper<T>(id, serializer)).name("collect()")
    	.output(new DiscardingOutputFormat<T>()).name("collect() sink");
    JobExecutionResult res = getExecutionEnvironment().execute();
    ```
    I'm really not sure how to handle accumulators in detached programs. What we can do is:
    In `JobExecutionResult`, whenever an accumulator is requested, we check for the runtime
to be -1, which indicates job was exited in detached mode. Display  a warning to the user
on the Cli itself and indicate that the results of accumulators are available on the web frontend
[which will require writing them to a file on the job manager]. 
    One more thing: Since the detached mode is available on Yarn, it must throw the exception
too in case of `collect` or `print` calls. Is this the case? I can't check on a yarn cluster
myself.


> CLI: Missing option to submit jobs in detached mode
> ---------------------------------------------------
>
>                 Key: FLINK-2797
>                 URL: https://issues.apache.org/jira/browse/FLINK-2797
>             Project: Flink
>          Issue Type: Bug
>          Components: Command-line client
>    Affects Versions: 0.9, 0.10
>            Reporter: Maximilian Michels
>            Assignee: Sachin Goel
>             Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a standalone installation.
This has been requested by users who want to submit a job, get the job id, and later query
its status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message