flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Gévay <gga...@gmail.com>
Subject Re: FLINK-5734 : Code Generation for NormalizedKeySorter
Date Wed, 08 Feb 2017 18:16:12 GMT
Hello Till,

> Why did you decide to generate the code on the TMs?

If we generated on the client side, then we would need to serialize
instances of the generated classes when shipping the job to the TMs,
but we would really like to avoid serializing instances of the
generated classes.
In the other code generation effort [1] (which is for the serializers
and comparators), we tried to do the generation on the client side,
because the serializers and comparators get instantiated on the client
side. However, it turned out that serializing and shipping instances
of a generated class to an other machine is a really big mess: When
you want to deserialize an object on an other machine then you need to
be able to load the class of that object on that machine. However, the
classloaders on that machine don't know about the class that was
generated in the client machine. (We ended up including the source
code of the generated class into the serialization as a string, and
essentially redoing the generation in readObject.)


[1] https://github.com/apache/flink/pull/2211

2017-02-08 12:01 GMT+01:00 Till Rohrmann <trohrmann@apache.org>:
> Hi Pat, Serkan and Gabor,
> I really like your design document and the preliminary results look really
> good. Impressive! I think your document can be converted into a FLIP.
> I was wondering whether it makes sense (or not) to generate the code on the
> client side? Then we would not have to introduce the `TemplateManager`,
> which avoids increasing the complexity of the TM, and we wouldn't need to
> ship TypeInformation to the TMs (I know this already happens but initially
> the idea was that TypeInformation is only used in the pre-flight phase).
> Why did you decide to generate the code on the TMs?
> Cheers,
> Till
> On Tue, Feb 7, 2017 at 10:34 PM, Pattarawat Chormai <pat.chormai@gmail.com>
> wrote:
>> Hi all,
>> We’re working on FLINK-5734 : Implement code generation for
>> NormalizedKeySorter. We have just finished a design document that describes
>> how we’re going to proceed this implementation.
>> Because this implement will touch several important parts of Flink, so we
>> would appreciate if you could give us some feedback before starting
>> implementation.
>> Here is the link to the document : https://docs.google.com/document/d/
>> 1anGQhBn9qI0yqe7twVvrDIiym4U4gxalJkZzM4Ar4QM/edit?usp=sharing <
>> https://docs.google.com/document/d/1anGQhBn9qI0yqe7twVvrDIiym4U4g
>> xalJkZzM4Ar4QM/edit?usp=sharing>
>> Also, we would like to confirm that NormalizedKeySorter is always
>> instantiated at TaskManager, but we could not find any solution to verify
>> that yet. If you know how to do it, we would be very pleased to hear that.
>> Thank you in advance for your feedback.
>> Best regards,
>> Serkan, Gábor and Pat

View raw message