apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganelin, Ilya" <Ilya.Gane...@capitalone.com>
Subject Apache Beam Integration
Date Mon, 09 May 2016 18:25:06 GMT
Hello, all – Google has just published a new blog announcing the first complete integration
of an open source project (Apache Flink) with Apache Beam:

https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective

http://data-artisans.com/why-apache-beam/

Apache Beam is a unifying framework that allows users to leverage disparate streaming computational
frameworks such as Storm, DataFlow, Flink, or Spark using a single API. This integration demands
that the framework conform to the Beam programming model: http://vldb.org/pvldb/vol8/p1792-Akidau.pdf,
and provides the appropriate APIs.

While cumbersome, the benefit of integrating with Beam is tremendous. Since there is no single
framework that solves all streaming problems for all use cases, the ability to combine frameworks
at-will makes developing end-state applications much more straightforward. I believe that
many projects will choose to leverage Apache Beam to take advantage of this and if Apex does
not provide support Beam, it will fall behind, replaced by those frameworks that fit the easy-to-use
model of Beam.

If we become early adopters, we have a unique opportunity to become part of what will quite
possible become a very large community of users and to capitalize on the inherent name recognition
of Google to elevate the Apex project and expose it to many who would otherwise not be aware
of it.

I think integration with Beam can pair with the recent work on developing a high-level API
for Apex and is a natural evolution towards making Apex more accessible and more usable by
a broader technical community.

If there is compelling interest around making this effort a reality, I would love to get this
conversation started and work on translating this into a concrete plan of action.


________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message