flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@apache.org>
Subject Re: Translate one Java API operator to multiple common API operators?
Date Fri, 21 Nov 2014 12:18:14 GMT
correction, there is no map(unwrap) in the example I gave.

2014-11-21 13:15 GMT+01:00 Fabian Hueske <fhueske@apache.org>:

> This is possible and already done.
> For example, a key-selector-groupReduce is translated into sth like:
> map(keySelector)->groupReduce()->map(unwrap)
>
> Since, MapFunctions are quite lightweight, there shouldn't be major
> performance implications. Make sure you get the DOP of all operators right,
> though.
>
> 2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <viktor.rosenfeld@tu-berlin.de
> >:
>
>> Hi,
>>
>> Is it possible to implement one Java API operator using multiple Common
>> API
>> operators?
>>
>> I.e., turn something like
>>
>>    IN -> OP(aggregate) -> OUT
>>
>> into
>>
>>    IN -> Map1 -> Reduce -> Map2 -> OUT
>>
>> I would imagine that the method SingleInputOperator.translateToDataFlow
>> could just create a chain of operators, wire the input to the first
>> operator
>> (Map1) and return the last operator of the chain (Map2).
>>
>> Is there anything I need to be aware of with this approach, e.g., some
>> performance issues?
>>
>> Best,
>> Viktor
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
>> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message