From user-return-22909-archive-asf-public=cust-asf.ponee.io@flink.apache.org Fri Sep 14 16:56:36 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4EC99180647 for ; Fri, 14 Sep 2018 16:56:34 +0200 (CEST) Received: (qmail 37185 invoked by uid 500); 14 Sep 2018 14:56:06 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 37174 invoked by uid 99); 14 Sep 2018 14:56:06 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2018 14:56:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A4E74C1F70 for ; Fri, 14 Sep 2018 14:55:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.009 X-Spam-Level: ***** X-Spam-Status: No, score=5.009 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RDNS_NONE=3, SPF_HELO_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5BXSOzvKN9iZ for ; Fri, 14 Sep 2018 14:55:44 +0000 (UTC) Received: from baidu.com (unknown [220.181.50.185]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 03D695F3EE for ; Fri, 14 Sep 2018 14:55:44 +0000 (UTC) Received: from BC-MAIL-HUB02.internal.baidu.com (unknown [172.31.56.56]) by Forcepoint Email with ESMTPS id B2A3D891262C3; Fri, 14 Sep 2018 22:55:38 +0800 (CST) Received: from BC-MAIL-MBX05.internal.baidu.com ([169.254.1.150]) by BC-MAIL-HUB02.internal.baidu.com ([172.31.56.56]) with mapi id 14.03.0415.000; Fri, 14 Sep 2018 22:55:36 +0800 From: "Yuan,Youjun" To: Timo Walther , "user@flink.apache.org" Subject: =?gb2312?B?tPC4tDogQ29udmVyc2lvbiB0byByZWxhdGlvbmFsIGFsZ2VicmEgZmFpbGVk?= =?gb2312?Q?_to_preserve_datatypes?= Thread-Topic: Conversion to relational algebra failed to preserve datatypes Thread-Index: AdRMBUSdwGEx0PC7SSq75kXwsJ+xMP//yeIA//9f2tA= Date: Fri, 14 Sep 2018 14:55:35 +0000 Message-ID: References: <217bf7c5-26e0-35fb-f134-5eb4a6227e0c@apache.org> In-Reply-To: <217bf7c5-26e0-35fb-f134-5eb4a6227e0c@apache.org> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.31.62.13] x-baidu-bdmsfe-datecheck: 1_BC-MAIL-HUB02_2018-09-14 22:55:39:393 x-baidu-bdmsfe-viruscheck: BC-MAIL-HUB02_GRAY_Inside_WithoutAtta_2018-09-14 22:55:36:507 Content-Type: multipart/alternative; boundary="_000_EDEFA088BB0F0A478CCEFC2EFE9ABD67197C6B5DBCMAILMBX05inte_" MIME-Version: 1.0 --_000_EDEFA088BB0F0A478CCEFC2EFE9ABD67197C6B5DBCMAILMBX05inte_ Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 SGkgVGltbywNCg0KSSBiZWxpZXZlIGl0IGhhcyBzb21ldGhpbmcgdG8gZG8gd2l0aCB0aGUgbnVt YmVyIG9mIGZpZWxkIGluIHRoZSBzb3VyY2UuIEkgY2FuIHJlcHJvZHVjZSB0aGUgaXNzdWUgd2l0 aCB2ZXJ5IHNpbXBsZSBTUUwsIGxpa2U6DQpJTlNFUlQgSU5UTyBJVkNfQUxBUk1fUFVCTElDX09V VCBTRUxFQ1Qgcm93dGltZSBhcyB0cyBGUk9NIElWQ19BTEFSTV9QVUJMSUNfSU4NCldoZXJlIHJv d3RpbWUgYWN0dWFsbHkgcmVwcmVzZW50IHByb2N0aW1lIChJIHJlZ2lzdGVyZWQgaXQgYXMgcm93 dGltZS5wcm9jdGltZSkuDQoNCklmIEkgcmVtb3ZlIGEgZmV3IGZpZWxkcyBpbiB0aGUgc291cmNl LCB0aGVuIGV2ZXJ5dGhpbmcgZ29lcyBmaW5lLg0KDQpUaGFua3MNCllvdWp1bg0KDQq3orz+yMs6 IFRpbW8gV2FsdGhlciA8dHdhbHRockBhcGFjaGUub3JnPg0Kt6LLzcqxvOQ6IEZyaWRheSwgU2Vw dGVtYmVyIDE0LCAyMDE4IDk6MTcgUE0NCsrVvP7IyzogdXNlckBmbGluay5hcGFjaGUub3JnDQrW 98ziOiBSZTogQ29udmVyc2lvbiB0byByZWxhdGlvbmFsIGFsZ2VicmEgZmFpbGVkIHRvIHByZXNl cnZlIGRhdGF0eXBlcw0KDQpIaSwNCg0KY291bGQgeW91IG1heWJlIHBvc3QgdGhlIHF1ZXJ5IHRo YXQgY2F1c2VkIHRoZSBleGNlcHRpb24/IEkgZ3Vlc3MgdGhlIGV4Y2VwdGlvbiBpcyByZWxhdGVk IHRvIGEgdGltZSBhdHRyaWJ1dGUgWzFdIGZvciB0aGUgb3B0aW1pemVyIHRpbWUgYXR0cmlidXRl cyBhbmQgdGltZXN0YW1wcyBtYWtlIG5vIGRpZmZlcmVuY2UgaG93ZXZlciB0aGV5IGhhdmUgYSBz bGlnaHRseSBkaWZmZXJlbnQgZGF0YSB0eXBlIHRoYXQgbWlnaHQgaGF2ZSBjYXVzZWQgdGhlIGVy cm9yLiBJIHRoaW5rIGlzIGEgYnVnIHRoYXQgc2hvdWxkIGJlIGZpeGVkLCBvbmNlIHdlIGhhdmUg bW9yZSBjb250ZXh0Lg0KDQpSZWdhcmRzLA0KVGltbw0KDQoNClsxXSBodHRwczovL2NpLmFwYWNo ZS5vcmcvcHJvamVjdHMvZmxpbmsvZmxpbmstZG9jcy1tYXN0ZXIvZGV2L3RhYmxlL3N0cmVhbWlu Zy5odG1sI3RpbWUtYXR0cmlidXRlcw0KDQoNCkFtIDE0LjA5LjE4IHVtIDEwOjQ5IHNjaHJpZWIg WXVhbixZb3VqdW46DQpIaSwNCg0KSSBhbSBnZXR0aW5nIHRoZSBmb2xsb3dpbmcgZXJyb3Igd2hp bGUgc3VibWl0dGluZyBqb2IgdG8gYSBjbHVzdGVyLCB3aGljaCBzZWVtcyBmYWlsZWQgdG8gY29t cGFyZSAyIFJlbERhdGVUeXBlcywgdGhvdWdoIHRoZXkgc2VlbXMgaWRlbnRpY2FsIChmcm9tIHRo ZSBlcnJvciBtZXNzYWdlKSwgYW5kIGV2ZXJ5dGhpbmcgaXMgT0sgaWYgSSBydW4gaXQgbG9jYWxs eS4NCkkgZ3Vlc3MgY2FsY2l0ZSBmYWlsZWQgdG8gY29tcGFyZSB0aGUgZmlyc3QgZmllbGQgbmFt ZWQgdHMsIG9mIHR5cGUgVElNRVNUQU1QKDMpLCBiZWNhdXNlOg0KDQogICogICBJZiBJIGRvbqGv dCBzZWxlY3QgdHMsIHRoZW4gZXZlcnl0aGluZyBnb2VzIGZpbmUNCiAgKiAgIElmIEkgY2FzdCB0 cyB0byBvdGhlciB0eXBlLCBsaWtlIFNFTEVDVCBjYXN0KHRzIEFTIFRJTUVTVEFNUCksIHRoZW4g ZXZlcnl0aGluZyBpcyBmaW5lDQogICogICBJZiBJIHN3aXRjaCB0byBFdmVudFRpbWUsIHRoZSBp c3N1ZSBhbHNvIGdvZXMgYXdheS4gQ3VycmVudGx5IGl0oa9zIFByb2Nlc3NUaW1lDQoNCkkgYW0g dXNpbmcgRmxpbmsgMS40LCBhbmQgc3VibWl0dGluZyBqb2IgdG8gYSBzdGFuZGFsb25lIGNsdXN0 ZXIuDQoNCkJlbG93IGFyZSB0aGUgZXJyb3I6DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tDQpDYXVzZWQgYnk6IG9yZy5hcGFjaGUuZmxpbmsuY2xpZW50LnBy b2dyYW0uUHJvZ3JhbUludm9jYXRpb25FeGNlcHRpb246IFRoZSBwcm9ncmFtIGNhdXNlZCBhbiBl cnJvcjoNCmF0IG9yZy5hcGFjaGUuZmxpbmsuY2xpZW50LnByb2dyYW0uT3B0aW1pemVyUGxhbkVu dmlyb25tZW50LmdldE9wdGltaXplZFBsYW4oT3B0aW1pemVyUGxhbkVudmlyb25tZW50LmphdmE6 OTMpDQphdCBvcmcuYXBhY2hlLmZsaW5rLmNsaWVudC5wcm9ncmFtLkNsdXN0ZXJDbGllbnQuZ2V0 T3B0aW1pemVkUGxhbihDbHVzdGVyQ2xpZW50LmphdmE6MzM0KQ0KYXQgb3JnLmFwYWNoZS5mbGlu ay5ydW50aW1lLndlYm1vbml0b3IuaGFuZGxlcnMuSmFyQWN0aW9uSGFuZGxlci5nZXRKb2JHcmFw aEFuZENsYXNzTG9hZGVyKEphckFjdGlvbkhhbmRsZXIuamF2YTo3NikNCmF0IG9yZy5hcGFjaGUu ZmxpbmsucnVudGltZS53ZWJtb25pdG9yLmhhbmRsZXJzLkphclJ1bkhhbmRsZXIubGFtYmRhJGhh bmRsZUpzb25SZXF1ZXN0JDAoSmFyUnVuSGFuZGxlci5qYXZhOjY5KSAuLi4gOSBtb3JlDQpDYXVz ZWQgYnk6IGphdmEubGFuZy5Bc3NlcnRpb25FcnJvcjogQ29udmVyc2lvbiB0byByZWxhdGlvbmFs IGFsZ2VicmEgZmFpbGVkIHRvIHByZXNlcnZlIGRhdGF0eXBlczoNCnZhbGlkYXRlZCB0eXBlOg0K UmVjb3JkVHlwZShUSU1FU1RBTVAoMykgTk9UIE5VTEwgdHMsIFZBUkNIQVIoNjU1MzYpIENIQVJB Q1RFUiBTRVQgIlVURi0xNkxFIiBDT0xMQVRFICJJU08tODg1OS0xJGVuX1VTJHByaW1hcnkiIHVz ZXJJZCwgVkFSQ0hBUig2NTUzNikgQ0hBUkFDVEVSIFNFVCAiVVRGLTE2TEUiIENPTExBVEUgIklT Ty04ODU5LTEkZW5fVVMkcHJpbWFyeSIgYmF0Y2hJZCwgVkFSQ0hBUig2NTUzNikgQ0hBUkFDVEVS IFNFVCAiVVRGLTE2TEUiIENPTExBVEUgIklTTy04ODU5LTEkZW5fVVMkcHJpbWFyeSIgcHJvamVj dElkLCBWQVJDSEFSKDY1NTM2KSBDSEFSQUNURVIgU0VUICJVVEYtMTZMRSIgQ09MTEFURSAiSVNP LTg4NTktMSRlbl9VUyRwcmltYXJ5IiB2ZWhpY2xlSWQsIENIQVIoNSkgQ0hBUkFDVEVSIFNFVCAi VVRGLTE2TEUiIENPTExBVEUgIklTTy04ODU5LTEkZW5fVVMkcHJpbWFyeSIgTk9UIE5VTEwgZmll bGQwLCBWQVJDSEFSKDY1NTM2KSBDSEFSQUNURVIgU0VUICJVVEYtMTZMRSIgQ09MTEFURSAiSVNP LTg4NTktMSRlbl9VUyRwcmltYXJ5IiBydWxlMCwgRE9VQkxFIHRocmVzaG9sZDAsIERPVUJMRSBm aWVsZF92YWx1ZTAsIFZBUkNIQVIoNjU1MzYpIENIQVJBQ1RFUiBTRVQgIlVURi0xNkxFIiBDT0xM QVRFICJJU08tODg1OS0xJGVuX1VTJHByaW1hcnkiIGFsYXJtSWQpIE5PVCBOVUxMDQpjb252ZXJ0 ZWQgdHlwZToNClJlY29yZFR5cGUoVElNRVNUQU1QKDMpIE5PVCBOVUxMIHRzLCBWQVJDSEFSKDY1 NTM2KSBDSEFSQUNURVIgU0VUICJVVEYtMTZMRSIgQ09MTEFURSAiSVNPLTg4NTktMSRlbl9VUyRw cmltYXJ5IiB1c2VySWQsIFZBUkNIQVIoNjU1MzYpIENIQVJBQ1RFUiBTRVQgIlVURi0xNkxFIiBD T0xMQVRFICJJU08tODg1OS0xJGVuX1VTJHByaW1hcnkiIGJhdGNoSWQsIFZBUkNIQVIoNjU1MzYp IENIQVJBQ1RFUiBTRVQgIlVURi0xNkxFIiBDT0xMQVRFICJJU08tODg1OS0xJGVuX1VTJHByaW1h cnkiIHByb2plY3RJZCwgVkFSQ0hBUig2NTUzNikgQ0hBUkFDVEVSIFNFVCAiVVRGLTE2TEUiIENP TExBVEUgIklTTy04ODU5LTEkZW5fVVMkcHJpbWFyeSIgdmVoaWNsZUlkLCBDSEFSKDUpIENIQVJB Q1RFUiBTRVQgIlVURi0xNkxFIiBDT0xMQVRFICJJU08tODg1OS0xJGVuX1VTJHByaW1hcnkiIE5P VCBOVUxMIGZpZWxkMCwgVkFSQ0hBUig2NTUzNikgQ0hBUkFDVEVSIFNFVCAiVVRGLTE2TEUiIENP TExBVEUgIklTTy04ODU5LTEkZW5fVVMkcHJpbWFyeSIgcnVsZTAsIERPVUJMRSB0aHJlc2hvbGQw LCBET1VCTEUgZmllbGRfdmFsdWUwLCBWQVJDSEFSKDY1NTM2KSBDSEFSQUNURVIgU0VUICJVVEYt MTZMRSIgQ09MTEFURSAiSVNPLTg4NTktMSRlbl9VUyRwcmltYXJ5IiBhbGFybUlkKSBOT1QgTlVM TA0KLi4uDQphdCBvcmcuYXBhY2hlLmNhbGNpdGUuc3FsMnJlbC5TcWxUb1JlbENvbnZlcnRlci5j aGVja0NvbnZlcnRlZFR5cGUoU3FsVG9SZWxDb252ZXJ0ZXIuamF2YTo0NTEpDQphdCBvcmcuYXBh Y2hlLmNhbGNpdGUuc3FsMnJlbC5TcWxUb1JlbENvbnZlcnRlci5jb252ZXJ0UXVlcnkoU3FsVG9S ZWxDb252ZXJ0ZXIuamF2YTo1NjcpDQphdCBvcmcuYXBhY2hlLmZsaW5rLnRhYmxlLmNhbGNpdGUu RmxpbmtQbGFubmVySW1wbC5yZWwoRmxpbmtQbGFubmVySW1wbC5zY2FsYToxMDYpDQotLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCnRoYW5rcyBpbiBhZHZh bmNlLA0KeW91anVuDQoNCg0K --_000_EDEFA088BB0F0A478CCEFC2EFE9ABD67197C6B5DBCMAILMBX05inte_ Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable

Hi Timo,=

 

I believe it has so= mething to do with the number of field in the source. I can reproduce the i= ssue with very simple SQL, like:

INSERT INTO IVC_= ALARM_PUBLIC_OUT SELECT rowtime as ts FROM IVC_ALARM_PUBLIC_IN

Where rowtime actua= lly represent proctime (I registered it as rowtime.proctime).

 

If I remove a few f= ields in the source, then everything goes fine.

 

Thanks

Youjun

 

=B7=A2=BC=FE=C8=CB: Timo Walther <twalthr@apache.org>
=B7=A2=CB=CD=CA=B1=BC=E4: Friday, Septem= ber 14, 2018 9:17 PM
=CA=D5=BC=FE=C8=CB: user@flink.apache.or= g
=D6=F7=CC=E2: Re: Conversion to relation= al algebra failed to preserve datatypes

 

Hi,

could you maybe post the query that caused the exception? I guess the excep= tion is related to a time attribute [1] for the optimizer time attributes a= nd timestamps make no difference however they have a slightly different dat= a type that might have caused the error. I think is a bug that should be fixed, once we have more context.
Regards,
Timo


[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming.= html#time-attributes


Am 14.09.18 um 10:49 schrieb Yuan,Youjun:

Hi,

 

I am getting the following error while submitting jo= b to a cluster, which seems failed to compare 2 RelDateTypes, though they s= eems identical (from the error message), and everything is OK if I run it l= ocally.

I guess calcite failed to compare the first field na= med ts, of type TIMESTAMP(3), because:

  • If I don=A1=AFt select ts, then everything goes fine
    • <= li class=3D"MsoListParagraph" style=3D"margin-left:0in;mso-list:l0 level1 l= fo3">If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP), then ev= erything is fine
  • If I switch to EventTime, the issue a= lso goes away. Currently it=A1=AFs ProcessTime

 

I am using Flink 1.4, and submitting job to a standa= lone cluster.

 

Below are the error:

----------------------------------------------------= ---------------------------------------------------------------------------= ---------------------------------

Caused by: org.apac= he.flink.client.program.ProgramInvocationException: The program caused an e= rror:       

at org.apache.flink= .client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvi= ronment.java:93) 

at org.apache.flink= .client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334) = ;     

at org.apache.flink= .runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(Jar= ActionHandler.java:76) 

at org.apache.flink= .runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRu= nHandler.java:69) ... 9 more

Caused by: java.lan= g.AssertionError: Conversion to relational algebra failed to preserve datat= ypes:

validated type:

RecordType(TIMESTAM= P(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE= "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET = "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, = VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary&= quot; projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE = "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "= ;UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field= 0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary&= quot; rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACT= ER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" al= armId) NOT NULL

converted type:

RecordType(TIMESTAM= P(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE= "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET = "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, = VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary&= quot; projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE = "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "= ;UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field= 0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary&= quot; rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACT= ER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" al= armId) NOT NULL

...

at org.apache.calci= te.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451)=  

at org.apache.calci= te.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567) =       

at org.apache.flink= .table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)=

----------------------------------------------------= ---------------------------------------------------------------------------= ---------------------------------

 

thanks in advance,

youjun

 

--_000_EDEFA088BB0F0A478CCEFC2EFE9ABD67197C6B5DBCMAILMBX05inte_--