From user-zh-return-2461-archive-asf-public=cust-asf.ponee.io@flink.apache.org Thu Mar 19 02:31:11 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D13B218025F for ; Thu, 19 Mar 2020 03:31:10 +0100 (CET) Received: (qmail 68286 invoked by uid 500); 19 Mar 2020 02:31:09 -0000 Mailing-List: contact user-zh-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-zh@flink.apache.org Delivered-To: mailing list user-zh@flink.apache.org Received: (qmail 68274 invoked by uid 99); 19 Mar 2020 02:31:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Mar 2020 02:31:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id E288DC2487 for ; Thu, 19 Mar 2020 02:31:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.251 X-Spam-Level: X-Spam-Status: No, score=0.251 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Px3wS_onE-gV for ; Thu, 19 Mar 2020 02:31:07 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::733; helo=mail-qk1-x733.google.com; envelope-from=zhisheng2018@gmail.com; receiver= Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 7532F7E12C for ; Thu, 19 Mar 2020 02:31:06 +0000 (UTC) Received: by mail-qk1-x733.google.com with SMTP id j4so856085qkc.11 for ; Wed, 18 Mar 2020 19:31:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=jZMOvlqYU3E2i3qDqJqGZxqeinkN/7xTqXCz6Brdazc=; b=n9LtT2/ApnsV/BJB9qJBXT4Y+2qJYb1cQEmKYDGNDiOGk1X1ece/8jh0MAPJ+WvoGG eR7PFzPUbxIboaQT5sX042zH7kIrYsh3WI/1aDag/xh2LKl/Yk8S1KcR9YIH1PKEyTXj jlF22SwvMGzsIFmScS1gGF0oiHZlgyIL/hZ9PXTuJURibAaZFGmL0zhjqqX8m8Y9laFd VIIn92YDHxm6ihgFeTG8Hxp/CWh9WDgYmKVFctnMxZuS5aH+uoGmvg6Tftd2a/yjAoWo REomfua2BgOaM56t2yrLwRRJY4cWEhc4Jee1sDMDA+cZ6qujogmkgxgKlkFXSEJohbNZ Azqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=jZMOvlqYU3E2i3qDqJqGZxqeinkN/7xTqXCz6Brdazc=; b=DEmFKfmGjjRAw5p/3Vhb2RMq2h4mnWhHs7pYOo0foHyqJVchJFAsGy7rlg2VcWICTP luATanX1qotQYzAXiusvrjTndgO3VVQe/4zCcrwRQ1jNJK8L2VG16nBqa3tVyegyP1OO aExbKyc/WTgGqmVwiytN30n/YnA13gAqBOke7oYsny/2VGuci7xHONkdp8fd0AJz+mTE mwmIFrw4dV5C/Zik89F8TaSqvU0Vr6Ws1fg3DcKD0C3FiYtQhFltYtp+HyH8MVitJ3ik WOF7FaqRYHOOUxb9VE5CKKqRntzUrCf1naz5HFE2XjL4CRzJvhgRoqVEJRyPUy6wfWgD URgg== X-Gm-Message-State: ANhLgQ3ueCtTRngOjae+LSWKMto/k+sf2HDYSXXkOpHriNvpq/TWP45p eFZiUvtKFLOjTcrNrUJkJKtH52gTLB5YCtJgxHZOMxgq X-Google-Smtp-Source: ADFU+vu//xyqVi4N1ZD+OIPLqlGZasjLcKsCvM00ffWFLW8Em6tKWcGH0pl699Q5mDj9l0LufxHkXiNnoYqlPcbQH4w= X-Received: by 2002:a37:3c9:: with SMTP id 192mr952937qkd.330.1584585064937; Wed, 18 Mar 2020 19:31:04 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: zhisheng Date: Thu, 19 Mar 2020 10:30:53 +0800 Message-ID: Subject: Re: Field types of query result and registered TableSink [Result] do not match To: user-zh Content-Type: multipart/alternative; boundary="000000000000e97ba105a12bf518" --000000000000e97ba105a12bf518 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable hi, Jark =E6=88=91=E5=8F=AA=E6=98=AF=E4=BD=BF=E7=94=A8=E4=BA=86 flink-jdbc =E8=BF=99= =E4=B8=AA connector=EF=BC=8C=E5=8F=91=E4=B8=8B=E6=88=91=E6=9C=AC=E5=9C=B0= =E6=B5=8B=E8=AF=95=E7=9A=84 DDL =E5=92=8C SQL =E5=A6=82=E4=B8=8B=EF=BC=9A String ddlSource =3D "CREATE TABLE test (\n" + " yidun_score numeric(5, 2)\n" + ") WITH (\n" + " 'connector.type' =3D 'kafka',\n" + " 'connector.version' =3D '0.11',\n" + " 'connector.topic' =3D 'test',\n" + " 'connector.startup-mode' =3D 'latest-offset',\n" + " 'connector.properties.zookeeper.connect' =3D 'localhost:2181',\n" + " 'connector.properties.bootstrap.servers' =3D 'localhost:9092',\n" + " 'format.type' =3D 'json'\n" + ")"; String ddlSink =3D "CREATE TABLE test_aggregate (\n" + " yidun_score numeric(5, 2)\n" + ") WITH (\n" + " 'connector.type' =3D 'jdbc',\n" + " 'connector.driver' =3D 'org.postgresql.Driver',\n" + " 'connector.url' =3D 'jdbc:postgresql://localhost:3600/test',\n" + " 'connector.table' =3D 'test_aggregate', \n" + " 'connector.username' =3D 'admin', \n" + " 'connector.password' =3D '1234546',\n" + " 'connector.write.flush.max-rows' =3D '1' \n" + ")"; String sql =3D "insert into test_aggregate select yidun_score from test"; blinkStreamTableEnv.sqlUpdate(ddlSource); blinkStreamTableEnv.sqlUpdate(ddlSink); blinkStreamTableEnv.sqlUpdate(sql); =E6=B2=A1=E6=9C=89=E8=87=AA=E5=AE=9A=E4=B9=89=E8=BF=87 TableSink Jark Wu =E4=BA=8E2020=E5=B9=B43=E6=9C=8819=E6=97=A5=E5= =91=A8=E5=9B=9B =E4=B8=8A=E5=8D=889:43=E5=86=99=E9=81=93=EF=BC=9A > Hi zhisheng, > > =E6=88=91=E7=8C=9C=E6=B5=8B=E4=BD=A0=E7=9A=84 TableSink =E7=9A=84=E5=AE= =9E=E7=8E=B0=E7=94=A8=E4=BA=86 legacy type, i.e. TypeInformation=EF=BC=8C = =E8=80=8C=E4=B8=8D=E6=98=AF DataType=E3=80=82 > legacy type =E6=98=AF=E6=97=A0=E6=B3=95=E8=A1=A8=E8=BE=BE=E7=B2=BE=E5=BA= =A6=E7=9A=84=EF=BC=8C=E6=89=80=E4=BB=A5=E5=8F=AA=E8=83=BD mapping =E5=88=B0= =E9=BB=98=E8=AE=A4=E7=9A=84 38, 18 =E4=B8=8A=E3=80=82 > =E8=BF=99=E6=98=AF=E6=A1=86=E6=9E=B6=E5=81=9A=E7=9A=84=E4=B8=80=E4=B8=AA= =E5=90=88=E6=B3=95=E6=80=A7=E6=A0=A1=E9=AA=8C=E3=80=82 > > Best, > Jark > > On Thu, 19 Mar 2020 at 09:33, zhisheng wrote: > > > hi, Jark > > > > =E6=88=91=E5=88=9A=E4=BD=BF=E7=94=A8 1.10.0 =E6=B5=8B=E8=AF=95=EF=BC=8C= =E6=8A=A5=E9=94=99=E5=BC=82=E5=B8=B8=E5=A6=82=E4=B8=8B=EF=BC=9A > > > > Exception in thread "main" > org.apache.flink.table.api.ValidationException: > > Type DECIMAL(5, 2) of table field 'yidun_score' does not match with the > > physical type LEGACY('DECIMAL', 'DECIMAL') of the 'yidun_score' field o= f > > the TableSink consumed type. > > at > > > > > org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogical= TypeCompatible$4(TypeMappingUtils.java:164) > > at > > > > > org.apache.flink.table.utils.TypeMappingUtils$1.visit(TypeMappingUtils.ja= va:265) > > at > > > > > org.apache.flink.table.utils.TypeMappingUtils$1.visit(TypeMappingUtils.ja= va:254) > > at > > > > > org.apache.flink.table.types.logical.LegacyTypeInformationType.accept(Leg= acyTypeInformationType.java:102) > > at > > > > > org.apache.flink.table.utils.TypeMappingUtils.checkIfCompatible(TypeMappi= ngUtils.java:254) > > at > > > > > org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCom= patible(TypeMappingUtils.java:160) > > at > > > > > org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$validateLogi= calPhysicalTypesCompatible$1.apply$mcVI$sp(TableSinkUtils.scala:287) > > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) > > at > > > > > org.apache.flink.table.planner.sinks.TableSinkUtils$.validateLogicalPhysi= calTypesCompatible(TableSinkUtils.scala:280) > > at > > > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(Pl= annerBase.scala:194) > > at > > > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(Pl= annerBase.scala:190) > > at scala.Option.map(Option.scala:146) > > at > > > > > org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(Plan= nerBase.scala:190) > > at > > > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(Pl= annerBase.scala:150) > > at > > > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(Pl= annerBase.scala:150) > > at > > > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca= la:234) > > at > > > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca= la:234) > > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234= ) > > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > > at > > > > > org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBa= se.scala:150) > > at > > > > > org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableE= nvironmentImpl.java:682) > > at > > > > > org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlUpdate(TableE= nvironmentImpl.java:495) > > at com.zhisheng.sql.blink.stream.example.Test.main(Test.java:49) > > Caused by: org.apache.flink.table.api.ValidationException: Legacy decim= al > > type can only be mapped to DECIMAL(38, 18). > > ... 26 more > > > > =E7=9C=8B=E6=8A=A5=E9=94=99=E6=97=A5=E5=BF=97=E5=83=8F=E6=98=AF=E5=8F= =AA=E8=83=BD=E4=BD=BF=E7=94=A8 DECIMAL(38, 18)=EF=BC=8C=E4=B8=8D=E7=84=B6= =E6=98=A0=E5=B0=84=E4=BC=9A=E5=A4=B1=E8=B4=A5=EF=BC=8C=E7=84=B6=E5=90=8E=E6= =88=91=E5=B0=86=E8=A1=A8=E5=AE=9A=E4=B9=89=E6=88=90 DECIMAL(38, 18) =E7=B1= =BB=E5=9E=8B=EF=BC=8C=E5=B0=B1=E4=B8=8D=E6=8A=A5=E9=94=99=E4=BA=86=E3=80=82 > > > > =E7=9C=8B=E4=BA=86=E4=B8=8B=E6=BA=90=E7=A0=81=E6=98=AF TypeMappingUtils= =E7=B1=BB=E4=B8=AD=E7=9A=84 checkIfCompatible =E5=81=9A=E4=BA=86=E6=A0=A1= =E9=AA=8C > > > > http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2020-03-19-012349.png > > > > =E7=9C=8B=E4=BA=86=E4=B8=8B=E8=BF=99=E4=B8=AA private =E6=96=B9=E6=B3= =95=E7=9A=84=E4=B8=8A=E5=B1=82=E6=98=AF checkPhysicalLogicalTypeCompatible = =E6=96=B9=E6=B3=95=EF=BC=8C=E8=BF=99=E4=B8=AA=E6=96=B9=E6=B3=95=E7=9A=84=E6= =B3=A8=E9=87=8A=E8=AF=B4=EF=BC=9A > > > > Checks whether the given physical field type and logical field type are > > compatible at the edges of the table ecosystem. Types are still > compatible > > if the physical type is a legacy decimal type (converted from > > Types#BIG_DEC) and the logical type is DECIMAL(38, 18). This is to > support > > legacy TypeInformation for TableSource and TableSink. > > > > =E7=9C=8B=E8=B5=B7=E6=9D=A5=E5=83=8F=E6=98=AF=E5=9C=A8=E5=85=BC=E5=AE= =B9=E6=97=A7=E7=9A=84 TypeInformation > > > > zhisheng =E4=BA=8E2020=E5=B9=B43=E6=9C=8819=E6= =97=A5=E5=91=A8=E5=9B=9B =E4=B8=8A=E5=8D=888:31=E5=86=99=E9=81=93=EF=BC=9A > > > > > =E5=AF=B9=E7=9A=84=EF=BC=8C=E8=BF=99=E6=98=AF=E7=94=9F=E4=BA=A7=E7=9A= =84=E4=B8=80=E4=B8=AA 1.9 blink planner =E4=BD=9C=E4=B8=9A=EF=BC=8C=E4=BB= =8A=E5=A4=A9=E6=88=91=E6=B5=8B=E8=AF=95=E4=B8=80=E4=B8=8B 1.10 blink planne= r > > > > > > Jark Wu =E4=BA=8E2020=E5=B9=B43=E6=9C=8818=E6=97= =A5=E5=91=A8=E4=B8=89 =E4=B8=8B=E5=8D=8811:47=E5=86=99=E9=81=93=EF=BC=9A > > > > > >> Hi zhisheng, > > >> > > >> =E4=BD=A0=E7=94=A8=E7=9A=84=E6=98=AF1.9=E5=90=97=EF=BC=9F =E8=AF=95= =E8=BF=87 1.10.0 blink planner =E4=B9=88=EF=BC=9F > > >> > > >> On Wed, 18 Mar 2020 at 22:21, zhisheng > wrote: > > >> > > >> > hi, all > > >> > > > >> > =E6=88=91=E5=AE=9A=E4=B9=89=E7=9A=84=E4=B8=80=E4=B8=AA=E8=A1=A8=E7= =9A=84=E4=B8=80=E4=B8=AA=E5=AD=97=E6=AE=B5=EF=BC=88yidun_score=EF=BC=89=E6= =98=AF numeric(5,2) =E7=B1=BB=E5=9E=8B=EF=BC=8C=E5=86=99=E5=85=A5 PostgreSQ= L =E7=9A=84 DDL > > >> yidun_score > > >> > =E5=AD=97=E6=AE=B5=E4=B9=9F=E6=98=AF=E5=AE=9A=E4=B9=89=E7=9A=84 nu= meric(5,2) =E7=B1=BB=E5=9E=8B=EF=BC=8C=E7=BB=93=E6=9E=9C=E4=BC=9A=E6=8A=A5= =E5=BC=82=E5=B8=B8=E3=80=82 > > >> > > > >> > org.apache.flink.client.program.ProgramInvocationException: The ma= in > > >> method > > >> > caused an error: Field types of query result and registered > TableSink > > >> > [Result] do not match. > > >> > Query result schema: [user_new_id: Long, total_credit_score: > Integer, > > >> > total_order_count: Integer, loss_total_order_count: Integer, > > >> yidun_score: > > >> > BigDecimal, is_delete: Boolean] > > >> > TableSink schema: [user_new_id: Long, total_credit_score: > Integer, > > >> > total_order_count: Integer, loss_total_order_count: Integer, > > >> yidun_score: > > >> > BigDecimal, is_delete: Boolean] > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedPr= ogram.java:593) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForE= xecution(PackagedProgram.java:438) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan= (OptimizerPlanEnvironment.java:83) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(Packa= gedProgramUtils.java:80) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(Packa= gedProgramUtils.java:122) > > >> > at > > >> > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:227) > > >> > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:20= 5) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:= 1010) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1= 083) > > >> > at java.security.AccessController.doPrivileged(Native Method) > > >> > at javax.security.auth.Subject.doAs(Subject.java:422) > > >> > at > > >> > > > >> > > > >> > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .java:1698) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(Hadoop= SecurityContext.java:41) > > >> > at > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > > >> > Caused by: org.apache.flink.table.api.ValidationException: Field > types > > >> of > > >> > query result and registered TableSink [Result] do not match. > > >> > Query result schema: [user_new_id: Long, total_credit_score: > Integer, > > >> > total_order_count: Integer, loss_total_order_count: Integer, > > >> yidun_score: > > >> > BigDecimal, is_delete: Boolean] > > >> > TableSink schema: [user_new_id: Long, total_credit_score: > Integer, > > >> > total_order_count: Integer, loss_total_order_count: Integer, > > >> yidun_score: > > >> > BigDecimal, is_delete: Boolean] > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.table.planner.sinks.TableSinkUtils$.validateSink(TableSi= nkUtils.scala:69) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(Pl= annerBase.scala:179) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(Pl= annerBase.scala:178) > > >> > at scala.Option.map(Option.scala:146) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(Plan= nerBase.scala:178) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(Pl= annerBase.scala:146) > > >> > at > > >> > > > >> > > > >> > > > org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(Pl= annerBase.scala:146) > > >> > at > > >> > > > >> > > > >> > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca= la:234) > > >> > at > > >> > > > >> > > > >> > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.sca= la:234) > > >> > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > > >> > > > >> > =E6=88=91=E5=90=8E=E9=9D=A2=E6=8A=8A numeric(5,2) =E7=B1=BB=E5=9E= =8B=E9=83=BD=E6=94=B9=E6=88=90 double=EF=BC=8C=E7=BB=93=E6=9E=9C=E5=8F=91= =E7=8E=B0=E5=B0=B1=E4=B8=8D=E6=8A=A5=E5=BC=82=E5=B8=B8=E4=BA=86=EF=BC=8C=E5= =8F=AF=E4=BB=A5=E6=AD=A3=E5=B8=B8=E5=B0=86=E6=95=B0=E6=8D=AE=E5=86=99=E5=85= =A5 PG=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93=E8=BF=99=E4=B8=AA case > > =E6=98=AF=E4=B8=8D=E6=98=AF=E4=B8=80=E4=B8=AA > > >> > bug=EF=BC=9F > > >> > > > >> > > > > > > --000000000000e97ba105a12bf518--