Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 49664200C4B for ; Mon, 20 Mar 2017 11:41:41 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 44CF6160B81; Mon, 20 Mar 2017 10:41:41 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4045A160B76 for ; Mon, 20 Mar 2017 11:41:40 +0100 (CET) Received: (qmail 47553 invoked by uid 500); 20 Mar 2017 10:41:39 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 47540 invoked by uid 99); 20 Mar 2017 10:41:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Mar 2017 10:41:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 4DB701AFBB1 for ; Mon, 20 Mar 2017 10:41:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id BMPtDe06tTBk for ; Mon, 20 Mar 2017 10:41:35 +0000 (UTC) Received: from mail-qt0-f174.google.com (mail-qt0-f174.google.com [209.85.216.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5A0085FD00 for ; Mon, 20 Mar 2017 10:41:35 +0000 (UTC) Received: by mail-qt0-f174.google.com with SMTP id x35so102717335qtc.2 for ; Mon, 20 Mar 2017 03:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Rzd55dPgLP77ei45kOzKVFV4LMnTjoaYcxBasbIDAz0=; b=s9SQBEA9mtfNXtXsFj9tIKk39H55nHl477gG2qlFPYDeOKXfkHAGHf8+A+zpSyzafe XrGvAK7mAA8Jarz7jhEq1yRlNhyMieG51CXe8Lfd74J0v05vgwCBh1aQBfsoZ4yCUsAe KqVcfXaaSYxWWi9tVzES6b3mSItR/8Q0BEG3l90d/bx/cpP1t68QRDiuqvG/ifbvd9pj LmGR4jg9k1yv9pX4StAdfyEMx8aee0olerbjjLmg37FSQJVV7xBTP2Sn2MjZXywDtUJF V0NtRyZLtnFhR811goIDGEjqhccVhEAjkh4xSnJmDOmH/14Hhi48iBE3a5eg2bN0Y+UO jLtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Rzd55dPgLP77ei45kOzKVFV4LMnTjoaYcxBasbIDAz0=; b=Uxgb2lGutVZuCh8SYTaqbE9J40n44SkH2o4b/lE55KXBHR12ZPYp8IByoVYF8UWctC yC5dah7BchXZZLVNhR4b+o0T9o4aDD1QUt6RQE0WsmCg7+RceIPum4QFibWzvw8YLaJz Uvp+yFBptZ+Y0MEbDxxHTFvGTydivJR4ODgmwIR9OzdfjN1siGYchlksukYrdfvJ7mkj YA9a9L+gHgB2hDKI4MSQZQyocf6ZwZtI0vIFF0Niz4dscR1TX5hJan/ucZrFO+vD6VRl OP2zocYja8wBvtVRx55Ni3/OtquNH3E96bGyyWmzOccseolNHqogeYxHPFQeoqgNnVy3 Bqng== X-Gm-Message-State: AFeK/H1iC8dQd520bq7iMhCx6QlWuJN9QeDMhgp72BYVF8f4S+WSosSLdvBCGm6mX/lKN107KN8/eMaU/Wch4w== X-Received: by 10.237.59.115 with SMTP id q48mr24401515qte.85.1490006489487; Mon, 20 Mar 2017 03:41:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.34.72 with HTTP; Mon, 20 Mar 2017 03:40:59 -0700 (PDT) In-Reply-To: References: From: Fabian Hueske Date: Mon, 20 Mar 2017 11:40:59 +0100 Message-ID: Subject: Re: SQL + flatten (or .*) quality docs location? To: user@flink.apache.org Content-Type: multipart/alternative; boundary=94eb2c0e5d9084e4ba054b272bc3 archived-at: Mon, 20 Mar 2017 10:41:41 -0000 --94eb2c0e5d9084e4ba054b272bc3 Content-Type: text/plain; charset=UTF-8 Hi Stu, thanks for reporting back. I tried to reproduce the "method flatten() not found" error but did not succeed. It would be great if you could open a JIRA issue and describe how to reproduce the problem. Thank you, Fabian 2017-03-17 16:42 GMT+01:00 Stu Smith : > Thank you! Just in case someone else stumbles onto this, I figured what > was giving me trouble. > The object I wanted to flattened happened to be null at times, at which > point it would error out and give some exception along the lines of: > > "method flatten() not found" > > (Sorry, I'll try to follow up with the actual trace to help people with > their searches later) > > which made it sound more like I was using it incorrectly altogether, > rather than that the object was null... I think even just letting NPE would > have been a little more helpful... > > I figured it out, btw, but trying equivalent programs in Flink and Esper > (doing an eval of both). Esper gave a clearer error, and then I went back > and fixed the flink program. > > Take care, > -stu > > On Thu, Mar 16, 2017 at 3:27 AM, Fabian Hueske wrote: > >> Hi Stu, >> >> there is only one page of documentation for the Table API and SQL [1]. >> I agree the structure could be improved and split into multiple pages. >> >> Regarding the flatting of a Pojo have a look at the "Built-In Functions" >> section [2]. >> If you select "SQL" and head to the "Value access functions", you'll find >> >> > tableName.compositeType.* : Converts a Flink composite type (such as >> Tuple, POJO, etc.) and all of its direct subtypes into a flat >> representation where every subtype is a separate field. >> >> >> The following program works returns the correct result: >> >> // POJO definition >> class MyPojo(var x: Int, var y: Int) { >> def this() = this(0, 0) >> } >> >> // SQL query >> val env = ExecutionEnvironment.getExecutionEnvironment >> val tEnv = TableEnvironment.getTableEnvironment(env, config) >> >> val ds = env.fromElements((0, new MyPojo(1, 2)), (1, new MyPojo(2, 3)), >> (2, new MyPojo(3, 4)) ) >> tEnv.registerDataSet("Pojos", ds, 'id, 'pojo) >> >> val result = tEnv.sql("SELECT id, Pojos.pojo.* FROM Pojos") // you need >> to include the table name to flatten a Pojo >> >> val results = result.toDataSet[Row].collect() >> println(results.mkString("\n")) >> >> // Result >> 0,1,2 >> 1,2,3 >> 2,3,4 >> >> Best, Fabian >> >> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/ >> dev/table_api.html >> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/ >> dev/table_api.html#built-in-functions >> >> 2017-03-15 21:31 GMT+01:00 Stu Smith : >> >>> The documentation seems to indicate that there is a flatten method >>> available in the sql language interface (in the table of available >>> methods), or, alternatively using the '*' character somehow (in the text >>> above the table). >>> >>> Yet I cannot flatten a POJO type, nor can I find any sufficient >>> documentation in the official docs, searching the mailing list via >>> markmail, looking through the examples in the source, or looking for >>> through the SQL tests in the source. >>> >>> Can someone point me to the correct location for some solid flink SQL >>> examples and docs? >>> >>> Take care, >>> -stu >>> >> >> > --94eb2c0e5d9084e4ba054b272bc3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Stu,

thanks for r= eporting back.
I tried to reproduce the "method flatten() not= found" error but did not succeed.

It would be great if y= ou could open a JIRA issue and describe how to reproduce the problem.
Thank you,
Fabian
2017-03-17 16:42 GMT+01:00 Stu Smith <stu26= code@gmail.com>:
Thank you! Just in case someone else= stumbles onto this, I figured what was giving me trouble.
The obj= ect I wanted to flattened happened to be null at times, at which point it w= ould error out and give some exception along the lines of:

&qu= ot;method flatten() not found"

(Sorry, I'll try= to follow up with the actual trace to help people with their searches late= r)

which made it sound more like I was using it inc= orrectly altogether, rather than that the object was null... I think even j= ust letting NPE would have been a little more helpful...

I fig= ured it out, btw, but trying equivalent programs in Flink and Esper (doing = an eval of both). Esper gave a clearer error, and then I went back and fixe= d the flink program.

Take care,
=C2=A0 -stu
=

On Thu, Mar 16, 2017 at 3:27 AM, Fabian Hueske <fhu= eske@gmail.com> wrote:
Hi Stu,

there is only one page of = documentation for the Table API and SQL [1].
I agree the structure= could be improved and split into multiple pages.

Regardi= ng the flatting of a Pojo have a look at the "Built-In Functions"= section [2].
If you select "SQL" and head to the &= quot;Value access functions", you'll find

> tableN= ame.compositeType.* : Converts a Flink composite type (such as Tuple= , POJO, etc.)=20 and all of its direct subtypes into a flat representation where every=20 subtype is a separate field.


The following program wo= rks returns the correct result:

// POJO defini= tion
class MyPojo(var x: Int, var y: Int) {
=C2=A0 def thi= s() =3D this(0, 0)
}

// SQL query
val en= v =3D ExecutionEnvironment.getExecutionEnvironment
val tEnv =3D Tab= leEnvironment.getTableEnvironment(env, config)

val ds =3D env.f= romElements((0, new MyPojo(1, 2)), (1, new MyPojo(2, 3)), (2, new MyPojo(3,= 4)) )
tEnv.registerDataSet("Pojos", ds, 'id, 'pojo)
val result =3D tEnv.sql("SELECT id, Pojos.pojo.* FROM Pojos"= ;) // you need to include the table name to flatten a Pojo

val resul= ts =3D result.toDataSet[Row].collect()
println(results.mkString(&qu= ot;\n"))

// Result
0,1,2
1,2,3
2,3,4

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink= -docs-release-1.2/dev/table_api.html
[2] https://ci.apache.org/projects/flink/fl= ink-docs-release-1.2/dev/table_api.html#built-in-functions

2= 017-03-15 21:31 GMT+01:00 Stu Smith <stu26code@gmail.com>:=
The documentation seem= s to indicate that there is a flatten method available in the sql language = interface (in the table of available methods), or, alternatively using the = '*' character somehow (in the text above the table).=C2=A0

Yet I cannot flatten a POJO type, nor= can I find any sufficient documentation in the official docs, searching th= e mailing list via markmail, looking through the examples in the source, or= looking for through the SQL tests in the source.
Can someone point me to the correct location for = some solid flink SQL examples and docs?=C2=A0

Take care,
=C2=A0 -stu



--94eb2c0e5d9084e4ba054b272bc3--