flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arvid Heise (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1228) Add REST Interface to JobManager
Date Fri, 09 Jan 2015 08:49:35 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270772#comment-14270772
] 

Arvid Heise commented on FLINK-1228:
------------------------------------

My original intent with this ticket was indeed simply a means to use HTTP
to trigger, monitor, and possibly terminate jobs similar to the WebHTFS
interface.

Ideally, Flink would even wrap all HTTP calls in a java class. In that way,
the application protocol used to exchange messages with the job manager is
completely transparent. For example, WebHTFS wraps the API through the
common FileSystem interface [1]. Without knowing any details, a user simply
switches to WebHDFS by specifying the webhdfs protocol in the fs
configuration of Hadoop.

[1]
http://www.caseyandgary.com/javadoc/hadoop/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.html

On Fri, Jan 9, 2015 at 12:59 AM, Henry Saputra (JIRA) <jira@apache.org>



> Add REST Interface to JobManager
> --------------------------------
>
>                 Key: FLINK-1228
>                 URL: https://issues.apache.org/jira/browse/FLINK-1228
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Arvid Heise
>
> For rolling out jobs to an external cluster, we currently have 3 choices:
> a) Manual submission with Web Interface
> b) Automatic/Manual submission with CLClient
> c) Automatic submission with custom client
> I propose to add a way to submit jobs automatically through a HTTP Rest Interface. Among
other benefits, this extension allows an automatic submission of jobs through a restrictive
proxy.
> Rough idea:
> The web interface would offer a REST entry point for example /jobs. POSTing to this entry
point allows the submission of a new job and returns the job URL. http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html
> GETting the job URL returns a small status.
> DELETING the job URL aborts the job.
> GETting on the /jobs returns a list of active and scheduled jobs.
> Since Flink already has a Jetty web server and uses Json for other services, the basic
extension should require low effort. It would help Flink to be used inside larger corporations
and align the interfaces with the other state-of-the-art MapReduce systems (s3, HDFS, HBase
all have HTTP interface).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message