apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yan <da...@datatorrent.com>
Subject High level API: Request for ideas
Date Wed, 23 Dec 2015 21:42:17 GMT
Hi fellow Apex developers:

Apex has a comprehensive API for constructing DAG topologies for streaming
applications, using operators, ports and streams.  But this may seem too
much for folks who just want to build simple applications, or just to learn
about Apex.  For example, when you compare the code to do word count in
Apex with Spark Streaming or Flink, Apex requires much more code.

Apex:
https://github.com/apache/incubator-apex-malhar/tree/master/demos/wordcount/src/main

Spark Streaming:
https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java

Flink:
https://github.com/apache/flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/wordcount/WordCount.java

Note that their Scala versions are even simpler to use.

The high-level requirements I have in mind is as follow:

1. A simple-to-use high-level API similar to what Spark Streaming and Flink
have. And from the high-level API, the Apex engine will construct the
actual DAG topology at launch time.

2. The first language we will support is Java, but we will also want to
support Scala and possibly Python at some point, so the high-level API
should make it easy for implementing bindings for at least these two
languages.

3. We should be able to use the high-level API in Apex App Package (apa)
file, so that dtcli can launch it just like a regular apa today.

Please provide your ideas and thoughts on this topic.

Thanks,

David

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message