flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@apache.org>
Subject Re: Translate one Java API operator to multiple common API operators?
Date Fri, 21 Nov 2014 12:15:49 GMT
This is possible and already done.
For example, a key-selector-groupReduce is translated into sth like:
map(keySelector)->groupReduce()->map(unwrap)

Since, MapFunctions are quite lightweight, there shouldn't be major
performance implications. Make sure you get the DOP of all operators right,
though.

2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <viktor.rosenfeld@tu-berlin.de>:

> Hi,
>
> Is it possible to implement one Java API operator using multiple Common API
> operators?
>
> I.e., turn something like
>
>    IN -> OP(aggregate) -> OUT
>
> into
>
>    IN -> Map1 -> Reduce -> Map2 -> OUT
>
> I would imagine that the method SingleInputOperator.translateToDataFlow
> could just create a chain of operators, wire the input to the first
> operator
> (Map1) and return the last operator of the chain (Map2).
>
> Is there anything I need to be aware of with this approach, e.g., some
> performance issues?
>
> Best,
> Viktor
>
>
>
> --
> View this message in context:
> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
> archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message