flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From schul...@informatik.hu-berlin.de
Subject Scala Code Generation
Date Tue, 13 Oct 2015 14:06:29 GMT

I am currently working on a compilation unit translating AsterixDB's AQL
into runnable Scala code for Flink's Scala API. During code generation I
discovered some things that are quite hard to work around. I am still
working with Flink version 0.8, so some of the problems I have might
already be fixed in 0.9 and if so please tell me.

First, whenever a record gets projected down to only a single field (e.g.
by a map or reduce function) it is no longer considered a record, but a
variable of the type of that field. If afterwards I want to apply
additional functions like .sum(0) I get an error message like

"Aggregating on field positions is only possible on tuple data types."

This is the same for all functions (like write or join) as the "record" is
no longer considered a dataset.

Second, I found that records longer than 22 fields are not supported.
Whenever I have a record that is longer than that I receive a build error

object Tuple23 is not a member of package scala

As I currently try to translate some TPCH queries the datasets might very
well get longer than that, although it is possible to work around that
using dedicated mapping of each dataset before joining them.

Thanks a lot for any help and best regards,
Max Schultze

View raw message