flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Changing how TypeComparators Work
Date Wed, 27 Aug 2014 15:02:50 GMT
Hi Guys,
while porting the Java API to Scala I'm noticing how complicated
things are because of how our TypeComparators work: 1) There is only
one type of comparator per TypeInformation which is created by the
TypeInformation. Therefore, our KeySelectors are not actually
implemented as comparators but as generated mappers that emit a
Tuple2, because you wouldn't for example be able to generate a
SelectorFunctionComparator for a TupleTypeInfo.  (There's also a lot
of magic going on with wrapping and unwrapping those tuples in Reduce,
Join, and CoGroup.) 2) Comparators cannot really interoperate, there
is special case code for the combinations that work. This will only
get worse when we properly introduce POJO types, which should work
well with tuple comparators and the other comparators.

My proposal is this: No more TypeComparator on a per type basis. Just
a generic comparator and PairComparator that work on Comparable. What
used to be TypeComparators become SelectionExtractors that return a
Comparable. Make Tuple comparable or add new ComparableTuple.  The
TupleSelectionExtractor would return a comparable tuple of the
appropriate length (same for POJOs). For Tuple extractors that operate
on only one field they would immediately return that field, without
wrapping it in a tuple. This would directly support our existing
KeySelector functions since the already return Comparable, when
returning a tuple in a key selector function this would be compatible
with a TupleSelectionExtractor (on the other join side, for example).

That's my idea. What do you think? I think the current state is not
maintainable, so we should do something quickly. :D

Cheers,
Aljoscha

Mime
View raw message