flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Tzoumas <ktzou...@apache.org>
Subject Re: Flink on Tez
Date Tue, 11 Nov 2014 08:15:36 GMT
Hi Henry, Tez has a local execution mode for debugging/experimenting etc:
http://tez.apache.org/localmode.html

On Tue, Nov 11, 2014 at 1:58 AM, Henry Saputra <henry.saputra@gmail.com>
wrote:

> HI Kostas,
>
> Since Tez underneath is using YARN so what does local execution means
> in this case?
>
> - Henry
>
> On Fri, Nov 7, 2014 at 10:03 AM, Kostas Tzoumas <ktzoumas@apache.org>
> wrote:
> > Hello Flink and Tez,
> >
> > I would like to point you to a first version of Flink running on
> > Tez. This is a Flink subproject (to be initially contributed
> > to flink-addons) that allows you to run unmodified Flink programs on
> > top of Apache Tez.
> >
> > You can get the code here:
> > https://github.com/ktzoumas/incubator-flink/tree/tez_support
> >
> > If you want to give it a spin, some basic instructions are here:
> >
> https://github.com/ktzoumas/incubator-flink/tree/tez_support/flink-addons/flink-tez
> >
> >
> > Be warned that this is still work in progress, so you may encounter
> > bugs, and this has not yet been optimized for performance.
> >
> > A few words on how it works and the motivation:
> >
> > The programs pass as usual through the Flink compiler and use the
> > Flink runtime operators (map, reduce, join, etc, including the Flink
> > facilities for sorting, hashing, etc). Instead of generating a Flink
> > distributed program (called "JobGraph" in Flink), we can now also
> > generate a Tez program (called "DAG" in Tez).
> >
> > I have been asked why would we want to do that, as Flink has its own
> > execution engine. Two reasons in my opinion.
> >
> > First, Tez follows design choices that are geared towards resource
> > elasticity, whereas the design choices behind Flink's engine are
> > geared more towards low latency querying and iterative
> > processing. Therefoere, the two engines can really complement each
> > other. Users can run their Flink programs in the engine that fits
> > better their use case and setup.
> >
> > Second, in Flink we have put a lot of effort in separating program
> > assembly with program execution and architecting the system in layers
> > (APIs, common API, compiler, data processing runtime, distributed
> > execution engine). The possibility to swap execution engines is a good
> > showcase of the benefits of such a layered architecture.
> >
> > Of course, trying it out and reporting bugs or contributing is very
> > welcome!
> >
> > Best,
> > Kostas
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message