flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: ReduceGroup fails on server
Date Fri, 30 Jan 2015 14:30:01 GMT
We have a bit of a divide in how we handle TypeSerializer and
TypeComparator: TypeSerializer does not handle duplication but relies
on outside code (RuntimeStatefulSerializerFactory) to perform the
duplication. TypeComparator does duplication itself. There is also the
RuntimeComparatorFactory. This, however, does not perform duplication
but simply hands out the same TypeComparator instance multiple times.
With comparators, the user of the comparator is responsible for
calling duplicate() on the comparator.

I started work on a branch where I add a duplicate() method to
TypeSerializer that simply return itself for stateless serializers and
does a deep copy if it is stateful. The TypeSerializers no longer have
method isStateful() and I replaced the two serializer factories by one
factory that always returns a "duplicate" of the serializer it holds.

I think we should consolidate the two approaches. So either let the
factories handle all the duplication or let the user of the
comparator/serializer always handle duplication. I think the former is
better, since the latter makes it very easy to forget to duplicate.

On Fri, Jan 30, 2015 at 11:34 AM, Aljoscha Krettek <aljoscha@apache.org> wrote:
> Maybe we can get rid of the serializer factories altogether. We could
> enhance the Serializers with a method duplicate() that does nothing
> for stateless serializers and does a deep copy for stateful
> serializers. This would also consolidate all the knowledge about
> stateful/stateless in one place.
>
> On Fri, Jan 30, 2015 at 11:25 AM, Aljoscha Krettek <aljoscha@apache.org> wrote:
>> Of course it doesn't work. The ClassLoader is declared transient in
>> the serializer factory, so it is Null once the factory has been
>> serialized/deserialized once. I open a Jira issue:
>> https://issues.apache.org/jira/browse/FLINK-1463
>>
>> On Thu, Jan 29, 2015 at 9:43 PM, Stephan Ewen <sewen@apache.org> wrote:
>>> Thanks for reporting this, Arvid. I remember that we saw something similar
>>> a while back. I will take a look at this later today (I am in PST zone
>>> now), can hopefully fix this.
>>>
>>> Greetings,
>>> Stephan

Mime
View raw message