From user-zh-return-56-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Feb 25 06:35:16 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9BDFA180626 for ; Mon, 25 Feb 2019 07:35:15 +0100 (CET) Received: (qmail 73750 invoked by uid 500); 25 Feb 2019 06:35:14 -0000 Mailing-List: contact user-zh-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user-zh@flink.apache.org Delivered-To: mailing list user-zh@flink.apache.org Received: (qmail 73738 invoked by uid 99); 25 Feb 2019 06:35:13 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Feb 2019 06:35:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 5F16BC599D for ; Mon, 25 Feb 2019 06:35:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.8 X-Spam-Level: *** X-Spam-Status: No, score=3.8 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id g7GdL9FmT7pa for ; Mon, 25 Feb 2019 06:35:11 +0000 (UTC) Received: from mail-yw1-f46.google.com (mail-yw1-f46.google.com [209.85.161.46]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 4C2845F5AA for ; Mon, 25 Feb 2019 06:35:11 +0000 (UTC) Received: by mail-yw1-f46.google.com with SMTP id x21so3233783ywx.11 for ; Sun, 24 Feb 2019 22:35:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=5nRmd5SJ1Alum7mjwYSCcsDJUiCZUx8c3FxCcfzU++c=; b=qoxcfUqw9My1GNOmAu7HYOPxg10Rr2W5thtKWpWFA/UWvnE6U0opWgJ+lyRucwFGSb uv8kZzYWctedRn58x/X/SvevwBLGiuLHCYN15zaxl3cZJrNk5Apan55KMikiGJJsZgl1 1ANJVMs2NlqlTvYn7KngRf/yDaLlHua5jDIh4aUQWfkmsKlND9z0LrO/uWnMTbUJboU5 PnjiDtvxDFqYXOGj2sBbIFajuJzbJ+uNtOXGVmave/7c1PhuZzmVHHSzP2DXNx+v640o NUpmYFRsj7/y/JyFp/83P6m/ZgpUEHSmVVSNe2XQ3Jx3umxtkSZ9WESXC5mAuWyQVcIp 2SXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=5nRmd5SJ1Alum7mjwYSCcsDJUiCZUx8c3FxCcfzU++c=; b=VbCxfVC65pY+TaEFI4n3ILADwhtpe/7i0aajKLmKTjV0ydJyLn0MAFNyQ+bmw5J2mw rg1j32OIQTExYAgt7oz2o0ucXILC3u/sstzjAnevzVerrxcgmbA7Zijt8iLqBZzdOcSw TzHegSp+dDcgg7Oj59uY6wh2MYEntkO1d3/LRwoHT9njujEa5foZsPR9hssUepOufvXa 8Q3FuERbI7ShMPtmPexyY8X3LyNcEXfSm32MlEhGAt5Z10EX58yj6komrtQR3DxMNe5v YxfKYhoD3J4psldNxBj/Gw8rXSTBDufBD6UzTf5lYaos4X4wjexjgFYukurq+TiqnqsE ur1g== X-Gm-Message-State: AHQUAubW3kk3MaQ5wDcqOCxnOF7MTNzsz/nZKeKFd+tVMt4CJPYR38gZ MxvMgoP3OBAX4vc5hHep8Sd4hiNh/b8+PRiHT14pot// X-Google-Smtp-Source: AHgI3IbljoYy0HXNWoEIXHoavXJ4ZtAbfsEGXd+P/Ir+yAKsAzApAtolHZo/SJdfNH0AN0Ea2oyz9V+bhgqGoqQfHq4= X-Received: by 2002:a81:f00c:: with SMTP id p12mr12744331ywm.242.1551076505311; Sun, 24 Feb 2019 22:35:05 -0800 (PST) MIME-Version: 1.0 References: <95d109d.6984.16913fd3e6f.Coremail.hongtao12310@163.com> <22f50617.abe6.16914d703cf.Coremail.hongtao12310@163.com> <32ea19d5.5db3.169225de83c.Coremail.hongtao12310@163.com> In-Reply-To: From: Becket Qin Date: Mon, 25 Feb 2019 14:34:54 +0800 Message-ID: Subject: =?UTF-8?B?UmU6IFJlOiBSZTogW0JsaW5rXXNxbCBjbGllbnQga2Fma2Egc2luayDlpLHotKU=?= To: user-zh@flink.apache.org Content-Type: multipart/alternative; boundary="0000000000001e5f4a0582b2244f" --0000000000001e5f4a0582b2244f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable @Kurt, =E8=BF=99=E4=B8=AA=E6=98=AF=E7=AC=A6=E5=90=88=E9=A2=84=E6=9C=9F=E7=9A=84=E3= =80=82=E4=B8=BA=E4=BA=86=E9=98=B2=E6=AD=A2=E5=92=8C=E7=94=A8=E6=88=B7code= =E4=B8=AD=E5=8F=AF=E8=83=BD=E7=9A=84Kafka=E4=BE=9D=E8=B5=96=E5=8F=91=E7=94= =9F=E5=86=B2=E7=AA=81=E3=80=82 On Mon, Feb 25, 2019 at 10:28 AM Kurt Young wrote: > kafka=E7=9A=84=E5=8C=85=E7=9C=8B=E8=B7=AF=E5=BE=84=E6=98=AFshade=E8=BF=87= =E7=9A=84=EF=BC=8C=E8=BF=99=E6=98=AF=E7=AC=A6=E5=90=88=E9=A2=84=E6=9C=9F=E7= =9A=84=E5=90=97=EF=BC=9F @Becket > > Best, > Kurt > > > On Mon, Feb 25, 2019 at 9:56 AM =E5=BC=A0=E6=B4=AA=E6=B6=9B wrote: > > > > > > > sql-client.sh =E7=9A=84=E5=90=AF=E5=8A=A8=E5=8F=82=E6=95=B0=E9=A6=96=E5= =85=88=E5=9C=A8classpath=E9=87=8C=E9=9D=A2=E4=BC=9A=E5=8C=85=E5=90=ABkafka= =E7=9B=B8=E5=85=B3=E7=9A=84jar =E5=8F=A6=E5=A4=96=E4=BC=9A=E6=9C=89--jar > > =E5=8C=85=E5=90=AB=E6=89=80=E6=9C=89connector=E7=9A=84jar > > > > > > =E8=BF=99=E4=BA=9Bjars=E5=9C=A8sql-client=E6=8F=90=E4=BA=A4job=E6=97=B6= =E5=80=99=E4=BC=9A=E4=B8=8A=E4=BC=A0=E5=88=B0cluster=E7=9A=84blob store =E4= =BD=86=E6=98=AF=E5=BE=88=E5=A5=87=E6=80=AA=E4=B8=BA=E5=95=A5=E6=89=BE=E4=B8= =8D=E5=88=B0 > > > > > > 00:00:06 /usr/lib/jvm/java-1.8.0-openjdk/bin/java > > -Dlog.file=3D/bigdata/flink-1.5.1/log/flink-root-sql-client-gpu06.log > > -Dlog4j.configuration=3Dfile:/bigdata/flink-1.5.1/conf/log4j-cli.proper= ties > > -Dlogback.configurationFile=3Dfile:/bigdata/flink-1.5.1/conf/logback.xm= l > > -classpath > > > /bigdata/flink-1.5.1/lib/flink-python_2.11-1.5.1.jar:/bigdata/flink-1.5.1= /lib/flink-shaded-hadoop2-uber-1.5.1.jar:/bigdata/flink-1.5.1/lib/log4j-1.2= .17.jar:/bigdata/flink-1.5.1/lib/slf4j-log4j12-1.7.7.jar:/bigdata/flink-1.5= .1/lib/flink-dist_2.11-1.5.1.jar::/bigdata/hadoop-2.7.5/etc/hadoop::/bigdat= a/flink-1.5.1/opt/connectors/kafka011/flink-connector-kafka-0.11_2.11-1.5.1= -sql-jar.jar:/bigdata/flink-1.5.1/opt/connectors/kafka010/flink-connector-k= afka-0.10_2.11-1.5.1-sql-jar.jar:/bigdata/flink-1.5.1/opt/connectors/kafka0= 9/flink-connector-kafka-0.9_2.11-1.5.1-sql-jar.jar:/bigdata/flink-1.5.1/opt= /connectors/kafka08/flink-connector-kafka-0.8_2.11-1.5.1.jar:/bigdata/flink= -1.5.1/opt/connectors/flink-hbase_2.11-1.5.1.jar:/bigdata/flink-1.5.1/opt/c= onnectors/flink-connector-hadoop-compatibility_2.11-1.5.1.jar:/bigdata/flin= k-1.5.1/opt/connectors/flink-connector-hive_2.11-1.5.1.jar:/bigdata/flink-1= .5.1/opt/sql-client/datanucleus-api-jdo-4.2.4.jar:/bigdata/flink-1.5.1/opt/= sql-client/javax.jdo-3.2.0-m3.jar:/bigdata/flink-1.5.1/opt/sql-client/datan= ucleus-core-4.1.17.jar:/bigdata/flink-1.5.1/opt/sql-client/datanucleus-rdbm= s-4.1.19.jar:/bigdata/flink-1.5.1/opt/sql-client/flink-sql-client-1.5.1.jar > > org.apache.flink.table.client.SqlClient embedded -d > > conf/sql-client-defaults.yaml --jar > > > /bigdata/flink-1.5.1/opt/connectors/kafka011/flink-connector-kafka-0.11_2= .11-1.5.1-sql-jar.jar > > --jar > > > /bigdata/flink-1.5.1/opt/connectors/kafka010/flink-connector-kafka-0.10_2= .11-1.5.1-sql-jar.jar > > --jar > > > /bigdata/flink-1.5.1/opt/connectors/kafka09/flink-connector-kafka-0.9_2.1= 1-1.5.1-sql-jar.jar > > --jar > > > /bigdata/flink-1.5.1/opt/connectors/kafka08/flink-connector-kafka-0.8_2.1= 1-1.5.1.jar > > --jar /bigdata/flink-1.5.1/opt/connectors/flink-hbase_2.11-1.5.1.jar > --jar > > > /bigdata/flink-1.5.1/opt/connectors/flink-connector-hadoop-compatibility_= 2.11-1.5.1.jar > > --jar > > /bigdata/flink-1.5.1/opt/connectors/flink-connector-hive_2.11-1.5.1.jar > > --jar /bigdata/flink-1.5.1/opt/sql-client/flink-sql-client-1.5.1.jar > > > > > > > > > > > > > > =E5=9C=A8 2019-02-22 19:32:18=EF=BC=8C"Becket Qin" =E5=86=99=E9=81=93=EF=BC=9A > > >=E8=83=BD=E4=B8=8D=E8=83=BD=E7=9C=8B=E4=B8=80=E4=B8=8B=E8=BF=90=E8=A1= =8Csql-client.sh=E7=9A=84=E8=BF=90=E8=A1=8C=E5=8F=82=E6=95=B0=E3=80=82=E5= =85=B7=E4=BD=93=E5=81=9A=E6=B3=95=E6=98=AF: > > > > > >=E8=BF=90=E8=A1=8Csql-client.sh > > >ps | grep sql-client > > > > > >=E6=9F=A5=E7=9C=8B=E4=B8=80=E4=B8=8B=E5=85=B6=E4=B8=AD=E6=98=AF=E4=B8= =8D=E6=98=AF=E6=9C=89=E8=BF=99=E4=B8=AA flink-connector-kafka-0.11 =E7=9A= =84 jar. > > > > > >Jiangjie (Becket) Qin > > > > > >On Fri, Feb 22, 2019 at 6:54 PM =E5=BC=A0=E6=B4=AA=E6=B6=9B wrote: > > > > > >> > > >> > > >> =E6=98=AF=E5=8C=85=E5=90=AB=E8=BF=99=E4=B8=AA=E7=B1=BB=E7=9A=84 > > >> > > >> > > >> jar -tf flink-connector-kafka-0.11_2.11-*.jar | grep Crc32C > > >> > > >> > > > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/Crc32C$1.c= lass > > >> > > >> > > > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/Crc32C$Che= cksumFactory.class > > >> > > >> > > > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/Crc32C$Jav= a9ChecksumFactory.class > > >> > > >> > > > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/Crc32C$Pur= eJavaChecksumFactory.class > > >> > > > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/Crc32C.cla= ss > > >> > > >> > > > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/PureJavaCr= c32C.class > > >> > > >> > > >> > > >> > > >> > > >> > > >> =E5=9C=A8 2019-02-22 18:03:18=EF=BC=8C"Zhenghua Gao" =E5=86=99=E9=81=93=EF=BC=9A > > >> >=E8=83=BD=E5=90=A6=E7=9C=8B=E4=B8=80=E4=B8=8B=E5=AF=B9=E5=BA=94=E7= =9A=84=E5=8C=85=E9=87=8C=E6=98=AF=E5=90=A6=E6=9C=89=E8=BF=99=E4=B8=AA=E7=B1= =BB=EF=BC=8C =E6=96=B9=E6=B3=95=E5=A6=82=E4=B8=8B(=E5=81=87=E8=AE=BE=E4=BD= =A0=E7=9A=84blink=E5=AE=89=E8=A3=85=E5=8C=85=E5=9C=A8 /tmp/blink): > > >> > > > >> >cd /tmp/blink/opt/connectors/kafka011 > > >> >jar -tf flink-connector-kafka-0.11_2.11-*.jar | grep Crc32C > > >> > > > >> >On Fri, Feb 22, 2019 at 2:56 PM =E5=BC=A0=E6=B4=AA=E6=B6=9B wrote: > > >> > > > >> >> > > >> >> > > >> >> =E5=A4=A7=E5=AE=B6=E5=A5=BD=EF=BC=81 > > >> >> > > >> >> > > >> >> =E6=88=91=E6=AD=A3=E5=9C=A8=E6=B5=8B=E8=AF=95Blink sql client kaf= ka sink connector =EF=BC=8C=E4=BD=86=E6=98=AF=E5=8F=91=E7=8E=B0=E5=86=99=E5= =85=A5=E5=A4=B1=E8=B4=A5=EF=BC=8C=E4=BB=A5=E4=B8=8B=E6=98=AF=E6=88=91=E7=9A= =84=E6=AD=A5=E9=AA=A4 > > >> >> > > >> >> > > >> >> =E7=8E=AF=E5=A2=83=E9=85=8D=E7=BD=AE > > >> >> blink standalone =E6=A8=A1=E5=BC=8F > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> 1. =E9=85=8D=E7=BD=AEenvironment =E5=90=AF=E5=8A=A8sql client > > >> >> > > >> >> > > >> >> 2. =E5=88=9B=E5=BB=BAkafka sink table > > >> >> CREATETABLEkafka_sink( > > >> >> messageKeyVARBINARY, > > >> >> messageValueVARBINARY, > > >> >> PRIMARYKEY(messageKey)) > > >> >> with( > > >> >> type=3D'KAFKA011', > > >> >> topic=3D'sink-topic', > > >> >> `bootstrap.servers`=3D'172.19.0.108:9092', > > >> >> retries=3D'3' > > >> >> ); > > >> >> > > >> >> > > >> >> 3. =E5=88=9B=E5=BB=BA=E6=9F=A5=E8=AF=A2=E8=AF=AD=E5=8F=A5=E5=B9= =B6=E6=89=A7=E8=A1=8C > > >> >> INSERT INTO kafka_sink > > >> >> SELECT CAST('123' AS VARBINARY) AS key, > > >> >> CAST(CONCAT_WS(',', 'HELLO', 'WORLD') AS VARBINARY) AS msg; > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> =E9=94=99=E8=AF=AF=E6=97=A5=E5=BF=97(from task executor log) > > >> >> > > >> >> > > >> >> =E4=B8=BB=E8=A6=81=E6=98=AF=E6=89=BE=E4=B8=8D=E5=88=B0kafka commo= n package=E4=B8=8B=E9=9D=A2=E7=9A=84=E4=B8=80=E4=B8=AA=E7=B1=BB=EF=BC=8C = =E4=BD=86=E6=98=AF=E5=90=AF=E5=8A=A8sql client =E6=97=B6=E5=80=99=E5=B7=B2= =E7=BB=8F=E6=8A=8Akafka > connector > > >> >> =E7=9B=B8=E5=85=B3=E7=9A=84jar=E5=8C=85=E5=8C=85=E6=8B=AC=E5=9C= =A8=E5=86=85 =E5=9C=A8=E6=8F=90=E4=BA=A4job=E6=97=B6=E5=80=99 =E4=B9=9F=E4= =BC=9A=E6=8A=8A=E8=BF=99=E4=BA=9Bjars =E5=92=8C > jobgraph=E4=B8=80=E5=B9=B6=E4=B8=8A=E4=BC=A0=E5=88=B0cluster=EF=BC=8C=E7= =90=86=E8=AE=BA=E4=B8=8A=E8=BF=99=E4=BA=9Bclass=E9=83=BD=E4=BC=9A=E8=A2=AB= =E5=8A=A0=E8=BD=BD > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> 2019-02-22 14:37:18,356 ERROR > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.common.utils.KafkaThrea= d > > >> >> - Uncaught exception in kafka-producer-network-thread | producer-= 1: > > >> >> java.lang.NoClassDefFoundError: > > >> >> > org/apache/flink/kafka011/shaded/org/apache/kafka/common/utils/Crc32C > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.common.record.DefaultRe= cordBatch.writeHeader(DefaultRecordBatch.java:468) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.common.record.MemoryRec= ordsBuilder.writeDefaultBatchHeader(MemoryRecordsBuilder.java:339) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.common.record.MemoryRec= ordsBuilder.close(MemoryRecordsBuilder.java:293) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.clients.producer.intern= als.ProducerBatch.close(ProducerBatch.java:391) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.clients.producer.intern= als.RecordAccumulator.drain(RecordAccumulator.java:485) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.clients.producer.intern= als.Sender.sendProducerData(Sender.java:254) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.clients.producer.intern= als.Sender.run(Sender.java:223) > > >> >> at > > >> >> > > >> > > > org.apache.flink.kafka011.shaded.org.apache.kafka.clients.producer.intern= als.Sender.run(Sender.java:162) > > >> >> at java.lang.Thread.run(Thread.java:748) > > >> >> Caused by: java.lang.ClassNotFoundException: > > >> >> > org.apache.flink.kafka011.shaded.org.apache.kafka.common.utils.Crc32C > > >> >> at > java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > >> >> at > > >> >> > > >> > > > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders= $ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:120) > > >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> -- > > >> >> Best Regards > > >> >> Hongtao > > >> >> > > >> >> > > >> > > > >> >-- > > >> >=E8=8B=A5=E6=89=B9=E8=A9=95=E7=84=A1=E8=87=AA=E7=94=B1=EF=BC=8C=E5= =89=87=E8=AE=9A=E7=BE=8E=E7=84=A1=E6=84=8F=E7=BE=A9=EF=BC=81 > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> -- > > >> Best Regards, > > >> HongTao > > >> > > >> > > > > > > > > > > > > > > > > -- > > Best Regards, > > HongTao > > > > > --0000000000001e5f4a0582b2244f--