Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C67591882E for ; Wed, 14 Oct 2015 09:42:20 +0000 (UTC) Received: (qmail 45434 invoked by uid 500); 14 Oct 2015 09:42:20 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 45361 invoked by uid 500); 14 Oct 2015 09:42:20 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 45352 invoked by uid 99); 14 Oct 2015 09:42:20 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2015 09:42:20 +0000 Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 240401A0098 for ; Wed, 14 Oct 2015 09:42:20 +0000 (UTC) Received: by wicge5 with SMTP id ge5so227906050wic.0 for ; Wed, 14 Oct 2015 02:42:18 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.194.8.227 with SMTP id u3mr2775336wja.38.1444815738924; Wed, 14 Oct 2015 02:42:18 -0700 (PDT) Received: by 10.28.139.69 with HTTP; Wed, 14 Oct 2015 02:42:18 -0700 (PDT) Received: by 10.28.139.69 with HTTP; Wed, 14 Oct 2015 02:42:18 -0700 (PDT) In-Reply-To: References: <55482.79.245.160.131.1444745189.squirrel@www2.informatik.hu-berlin.de> Date: Wed, 14 Oct 2015 11:42:18 +0200 Message-ID: Subject: Re: Scala Code Generation From: Till Rohrmann To: user@flink.apache.org Content-Type: multipart/alternative; boundary=047d7b5d9c07e279db05220d6077 --047d7b5d9c07e279db05220d6077 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable If you're using Scala, then you're bound to a maximum of 22 fields in a tuple, because the Scala library does not provide larger tuples. You could generate your own case classes which have more than the 22 fields, though. On Oct 14, 2015 11:30 AM, "Ufuk Celebi" wrote: > > > On 13 Oct 2015, at 16:06, schultze@informatik.hu-berlin.de wrote: > > > > Hello, > > > > I am currently working on a compilation unit translating AsterixDB's AQ= L > > into runnable Scala code for Flink's Scala API. During code generation = I > > discovered some things that are quite hard to work around. I am still > > working with Flink version 0.8, so some of the problems I have might > > already be fixed in 0.9 and if so please tell me. > > > > First, whenever a record gets projected down to only a single field (e.= g. > > by a map or reduce function) it is no longer considered a record, but a > > variable of the type of that field. If afterwards I want to apply > > additional functions like .sum(0) I get an error message like > > A workaround is to return Tuple1 for this. Then you can run the > aggregation. I think that the Tuple0 class has been added after 0.8 thoug= h. > > > "Aggregating on field positions is only possible on tuple data types." > > > > This is the same for all functions (like write or join) as the "record" > is > > no longer considered a dataset. > > What do you mean? At least in the current versions, the join projections > return a Tuple type as well. > > > Second, I found that records longer than 22 fields are not supported. > > Whenever I have a record that is longer than that I receive a build err= or > > as > > Flink=E2=80=99s Tuple classes go up to Tuple25. You can work around this = by using > a custom PoJo type, e.g. > > class TPCHRecord { > public int f0; > ... > public int f99; > } > > If possible, I would suggest to update to the latest 0.9 or the upcoming > 0.10 release. A lot of stuff has been fixed since 0.8. I think it will be > worth it. If you encounter any problems while doing this, feel free to as= k > here. :) > > =E2=80=93 Ufuk --047d7b5d9c07e279db05220d6077 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

If you're using Scala, then you're bound to a maximu= m of 22 fields in a tuple, because the Scala library does not provide large= r tuples. You could generate your own case classes which have more than the= 22 fields, though.

On Oct 14, 2015 11:30 AM, "Ufuk Celebi"= ; <uce@apache.org> wrote:

> On 13 Oct 2015, at 16:06, schultze@informatik.hu-berlin.de wrote:
>
> Hello,
>
> I am currently working on a compilation unit translating AsterixDB'= ;s AQL
> into runnable Scala code for Flink's Scala API. During code genera= tion I
> discovered some things that are quite hard to work around. I am still<= br> > working with Flink version 0.8, so some of the problems I have might > already be fixed in 0.9 and if so please tell me.
>
> First, whenever a record gets projected down to only a single field (e= .g.
> by a map or reduce function) it is no longer considered a record, but = a
> variable of the type of that field. If afterwards I want to apply
> additional functions like .sum(0) I get an error message like

A workaround is to return Tuple1<X> for this. Then you can run the ag= gregation. I think that the Tuple0 class has been added after 0.8 though.
> "Aggregating on field positions is only possible on tuple data ty= pes."
>
> This is the same for all functions (like write or join) as the "r= ecord" is
> no longer considered a dataset.

What do you mean? At least in the current versions, the join projections re= turn a Tuple type as well.

> Second, I found that records longer than 22 fields are not supported.<= br> > Whenever I have a record that is longer than that I receive a build er= ror
> as

Flink=E2=80=99s Tuple classes go up to Tuple25. You can work around this by= using a custom PoJo type, e.g.

class TPCHRecord {
=C2=A0 =C2=A0 public int f0;
=C2=A0 =C2=A0 ...
=C2=A0 =C2=A0 public int f99;
}

If possible, I would suggest to update to the latest 0.9 or the upcoming 0.= 10 release. A lot of stuff has been fixed since 0.8. I think it will be wor= th it. If you encounter any problems while doing this, feel free to ask her= e. :)

=E2=80=93 Ufuk
--047d7b5d9c07e279db05220d6077--