flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: KNIME Integration of Flink
Date Mon, 10 Nov 2014 15:23:35 GMT
Hi Arvid!

What you describe sounds like a great use case. I am not aware of an
integration of Flink with KNIME. Triggering progamms from other programs
should work through the Client & PackagedProgram classes-

 -
https://github.com/apache/incubator-flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/Client.java

 -
https://github.com/apache/incubator-flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java

Triggering programs through HTTPS REST should be possible with a thin layer
that wraps RPC calls / actor messages in HTTP requests.

Out-of-the-box images for Amazon do not exist with any current version of
the system. So far, elastic MapReduce with YARN has proved a simpler setup
than any VM we had prepared before. How would you skip the SSH key setup
with prepared images?

Greetings,
Stephan



On Mon, Nov 10, 2014 at 3:02 PM, Arvid Heise <arvid.heise@gmail.com> wrote:

> Dear Flinkler,
>
> For my current project, we want to outsource some performance critical
> parts of a complex KNIME workflow to Flink. Is there already a way to
> trigger a Flink workflow from KNIME? If not, we will probably provide a
> straight-forward way to execute Flink (Scala) programs from KNIME within
> this month. The overall goal is to upload the data from the KNIME workflow
> to s3, execute Flink on this data, and retrieve the output from s3.
>
> Since my employer (BfR) has a rather strict firewall setup, I will also add
> a feature request for a REST API of the job manager similar to s3 (
> http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-example.html),
> so
> that we can completely rely on HTTP(S).
>
> Lastly, are there any plans for providing out-of-the-box VMs for Amazon
> MapReduce? I saw the blog post
>
> https://flink.incubator.apache.org/news/2014/02/18/amazon-elastic-mapreduce-cloud-yarn.html
> , which would suffice, but still requires quite a bit manual work (with
> SSH, which does not work here).
>
> Feel free to split the topics into separate threads if you want to discuss
> them individually.
>
> Best,
>
> Arvid
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message