flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ouywl <ou...@139.com>
Subject Re: [Question] How to use different filesystem between checkpointdata and user data sink
Date Tue, 24 Dec 2019 06:42:32 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.6;
    }
    ul,ol{
        padding-left: 20px;
        list-style-position: inside;
    }
</style>
<div style = 'font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;'>
    <div ></div><div>
    <div>Hi Piotr:</div><div>&nbsp; &nbsp; &nbsp;As debug the code, I found The jobmanager classpath is not content “system-plugin.jar”, But when&nbsp;run `<span style="font-family: Menlo; font-size: 9pt;">configureFileSystems(</span><span style="font-family: Menlo; font-size: 9pt;">configuration</span><span style="font-family: Menlo; font-size: 9pt;">)` in `</span><span style="font-family: Menlo; font-size: 9pt;">ClusterEntrypoint.</span><span style="font-family: Menlo; font-size: 9pt;">startCluster</span><span class=" classDarkfont" style="font-family: Menlo; font-size: 9pt;">()`, It will&nbsp;<span style="font-style: italic;">initialize with</span>&nbsp;the FileSystem plugin in the method '</span><span style="font-family: Menlo; font-size: 9pt;">FileSystem.</span><span style="font-family: Menlo; font-size: 9pt; font-style: italic;">initialize</span><span style="font-family: Menlo; font-size: 9pt;">(configuration</span><span style="font-family: Menlo; font-size: 9pt;">, </span><span style="font-family: Menlo; font-size: 9pt;">PluginUtils.</span><span style="font-family: Menlo; font-size: 9pt; font-style: italic;">createPluginManagerFromRootFolder</span><span style="font-family: Menlo; font-size: 9pt;">(configuration))</span><span style="font-family: Menlo; font-size: 9pt;" class=" classDarkfont">;</span><font face="Menlo"><span style="font-size: 12px;">’</span><span style="font-size: 9pt;">, &nbsp;After that when blob getFilesystem,It will load FileSystem by schema, So it will call&nbsp;</span><span style="font-size: 12px;">“</span></font><span style="color: rgb(34, 34, 34); orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">MyFileSystemFactory</span><font face="Menlo"><span style="font-size: 12px;">”</span><span style="font-size: 9pt;">&nbsp;which&nbsp;</span><span style="font-size: 12px;">implements myself, and It have put core-site.xml,&nbsp;hfs-site.xml in&nbsp;</span></font>“system-plugin.jar”, It is no<b><font size="2" style="font-size: 12px;">t <span style="font-family: Menlo;">&nbsp;</span><span style="font-family: inherit; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(248, 249, 250);" class=" classDarkfont">Suitable for ‘</span></font></b><span style="color: rgb(34, 34, 34); orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">hdfs://slothTest/user/sloth/HA/</span><b><font style="font-size: 12px;"><span style="font-family: inherit; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(248, 249, 250);" class=" classDarkfont">’ 。</span></font></b></div><div><br></div><div>Full stack is :</div><div>&nbsp;&nbsp;<span style="color: rgb(34, 34, 34); font-family: sans-serif; font-size: 14.4px; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">Log Type: jobmanager.log</span></div><p style="margin: 0px; border: 0px; color: rgb(34, 34, 34); font-family: sans-serif; font-size: 14.4px; font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">Log Upload Time: Sun Dec 22 19:09:16 +0800 2019</p><p style="margin: 0px; border: 0px; color: rgb(34, 34, 34); font-family: sans-serif; font-size: 14.4px; font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">Log Length: 39600</p><pre style="margin-top: 0px; margin-bottom: 0px; border: 0px; color: rgb(34, 34, 34); font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">2019-12-22 19:09:12,305 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-12-22 19:09:12,307 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting YarnJobClusterEntrypoint (Version: 1.9.1, Rev:4d56de8, Date:30.09.2019 @ 11:32:19 CST)
2019-12-22 19:09:12,308 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: yarn
2019-12-22 19:09:12,790 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: sloth
2019-12-22 19:09:12,791 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.152-b16
2019-12-22 19:09:12,791 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 406 MiBytes
2019-12-22 19:09:12,791 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /usr/jdk64/jdk1.8.0_152
2019-12-22 19:09:12,792 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Hadoop version: 2.7.3
2019-12-22 19:09:12,792 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
2019-12-22 19:09:12,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms424m
2019-12-22 19:09:12,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx424m
2019-12-22 19:09:12,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -XX:+PrintGCDetails
2019-12-22 19:09:12,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -XX:+PrintGCDateStamps
2019-12-22 19:09:12,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xloggc:/home/sloth/hadoop/yarn/logs/application_1576548114502_0152/container_e70_1576548114502_0152_02_000001/jobmanager-gc.log
2019-12-22 19:09:12,793 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -XX:+UseGCLogFileRotation
2019-12-22 19:09:12,794 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -XX:NumberOfGCLogFiles=1
2019-12-22 19:09:12,794 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -XX:GCLogFileSize=1M
2019-12-22 19:09:12,794 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/home/sloth/hadoop/yarn/logs/application_1576548114502_0152/container_e70_1576548114502_0152_02_000001/jobmanager.log
2019-12-22 19:09:12,794 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlogback.configurationFile=file:logback.xml
2019-12-22 19:09:12,794 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:log4j.properties
2019-12-22 19:09:12,794 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments: (none)
2019-12-22 19:09:12,795 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: my-file-system-plugin-1.0-SNAPSHOT.jar:lib/flink-metrics-influxdb_2.11-sloth-flink-1.7.2.jar:lib/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:lib/flink-table-blink_2.12-1.9.1.jar:lib/flink-table_2.12-1.9.1.jar:lib/guava-19.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:log4j.properties:logback.xml:plugins/README.txt:flink.jar:flink-conf.yaml:job.graph::/usr/ndp/current/yarn_nodemanager/conf:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/hadoop-common-2.7.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/hadoop-nfs-2.7.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/hadoop-common-2.7.3-tests.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/hadoop-annotations-2.7.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/gson-2.2.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-io-2.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/activation-1.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/xz-1.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/asm-3.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/hadoop-lzo-0.4.20.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-net-3.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jettison-1.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/junit-4.11.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/paranamer-2.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/avro-1.7.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/hadoop-auth-2.7.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/common/lib/guava-11.0.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/hadoop-hdfs-2.7.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/hadoop-hdfs-2.7.3-tests.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/hadoop-lzo-0.4.20.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/ndp/3.3.0/yarn_nodemanager/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-api-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-common-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-common-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-registry-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-client-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/activation-1.1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/xz-1.0.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-math-2.2.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/guice-3.0.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/asm-3.2.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/hadoop-lzo-0.4.20.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/spark-2.1.0-yarn-shuffle.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/ndp/current/yarn_nodemanager/share/hadoop/yarn/lib/guava-11.0.2.jar
2019-12-22 19:09:12,796 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2019-12-22 19:09:12,798 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
2019-12-22 19:09:12,801 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - YARN daemon is running as: sloth Yarn client user obtainer: sloth
2019-12-22 19:09:12,805 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: env.java.opts.jobmanager, -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:&lt;LOG_DIR&gt;/jobmanager-gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=1M
2019-12-22 19:09:12,806 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: restart-strategy.failure-rate.max-failures-per-interval, 5000
2019-12-22 19:09:12,806 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.cluster-id, application_1576548114502_0152
2019-12-22 19:09:12,806 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.password, ******
2019-12-22 19:09:12,807 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.predefined-options, FLASH_SSD_OPTIMIZED
2019-12-22 19:09:12,807 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: FLINK_PLUGINS_DIR, /home/sloth/flink-1.9.1/plugins
2019-12-22 19:09:12,807 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.compaction.level.max-size-level-base, 536870912
2019-12-22 19:09:12,807 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.path.root, /sloth-flink
2019-12-22 19:09:12,807 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: sql.resource.external-buffer.memory.mb, 64
2019-12-22 19:09:12,808 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.connectTimeout, 100000
2019-12-22 19:09:12,808 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.timer-service.factory, ROCKSDB
2019-12-22 19:09:12,808 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.mem-table-flush-pending, true
2019-12-22 19:09:12,808 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.off-heap, true
2019-12-22 19:09:12,808 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporters, influxdb
2019-12-22 19:09:12,808 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-deletes-imm-mem-tables, true
2019-12-22 19:09:12,809 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.block.cache-size, 536870912
2019-12-22 19:09:12,809 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: restart-strategy.failure-rate.failure-rate-interval, 6 min
2019-12-22 19:09:12,809 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.class, org.apache.flink.metrics.influxdb.InfluxdbReporter
2019-12-22 19:09:12,809 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.compaction.level.use-dynamic-size, true
2019-12-22 19:09:12,809 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
2019-12-22 19:09:12,810 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.per-job-cluster.include-user-jar, FIRST
2019-12-22 19:09:12,810 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.compaction-pending, true
2019-12-22 19:09:12,810 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.username, flink-metrics
2019-12-22 19:09:12,810 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.tags, job-id@493,job-type@JAR,kernel-type@8,product@sloth,version@8,sys@sloth
2019-12-22 19:09:12,810 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.localdir, /mnt/dfs/1/rocksdb/
2019-12-22 19:09:12,811 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.incremental, true
2019-12-22 19:09:12,811 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.preallocate, false
2019-12-22 19:09:12,811 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.storageDir, hdfs://slothTest/user/sloth/HA/
2019-12-22 19:09:12,811 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-running-flushes, true
2019-12-22 19:09:12,811 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.port, 8091
2019-12-22 19:09:12,812 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-entries-active-mem-table, true
2019-12-22 19:09:12,812 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability.zookeeper.quorum, sloth-test0.dg.163.org:2181,sloth-test1.dg.163.org:2181,sloth-test2.dg.163.org:2181
2019-12-22 19:09:12,812 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-entries-imm-mem-tables, true
2019-12-22 19:09:12,812 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-running-compactions, true
2019-12-22 19:09:12,812 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: sql.resource.default.memory.mb, 64
2019-12-22 19:09:12,813 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend, rocksdb
2019-12-22 19:09:12,813 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.estimate-table-readers-mem, true
2019-12-22 19:09:12,813 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.estimate-pending-compaction-bytes, true
2019-12-22 19:09:12,813 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.host, sloth-tsdb0.dg.163.org,sloth-tsdb1.dg.163.org,sloth-tsdb2.dg.163.org
2019-12-22 19:09:12,813 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: sql.resource.sink.default.memory.mb, 128
2019-12-22 19:09:12,814 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.background-errors, true
2019-12-22 19:09:12,814 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.filter, (.*)numRecordsIn(.*),(.*)numRecordsOut(.*),(.*)Latency,(.*)records_consumed_rate,(.*)records_lag_max,(.*)fullRestarts(.*),(.*)uptime,(.*)numRunningJobs,(.*)numRegisteredTaskManagers,(.*)taskSlots(.*),(.*)Checkpoint(.*),(.*)Status_JVM(.*)
2019-12-22 19:09:12,814 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.db, flink-test-V2
2019-12-22 19:09:12,814 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.thread.num, 2
2019-12-22 19:09:12,814 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.maximum-failed-containers, 36000
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.network.numberOfBuffers, 4096
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: yarn.application-attempts, 36000
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-deletes-active-mem-table, true
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-snapshots, true
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: sql.resource.source.default.memory.mb, 128
2019-12-22 19:09:12,815 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.estimate-live-data-size, true
2019-12-22 19:09:12,816 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.fs.checkpointdir, hdfs://slothTest/user/sloth/sloth-fs-checkpoints/cpk/1_7
2019-12-22 19:09:12,816 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: restart-strategy.failure-rate.delay, 1 s
2019-12-22 19:09:12,816 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: blob.storage.directory, /mnt/dfs/0/yarn/local/blobJars/
2019-12-22 19:09:12,816 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.memory.size, 1024
2019-12-22 19:09:12,816 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.cur-size-active-mem-table, true
2019-12-22 19:09:12,816 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-immutable-mem-table, true
2019-12-22 19:09:12,817 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.size-all-mem-tables, true
2019-12-22 19:09:12,817 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.total-sst-files-size, true
2019-12-22 19:09:12,817 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.compaction.level.target-file-size-base, 33554432
2019-12-22 19:09:12,817 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.cur-size-all-mem-tables, true
2019-12-22 19:09:12,817 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.estimate-num-keys, true
2019-12-22 19:09:12,817 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: akka.framesize, 1048576000b
2019-12-22 19:09:12,818 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.actual-delayed-write-rate, true
2019-12-22 19:09:12,818 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: internal.cluster.execution-mode, DETACHED
2019-12-22 19:09:12,818 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.writeTimeout, 10000
2019-12-22 19:09:12,818 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: high-availability, zookeeper
2019-12-22 19:09:12,818 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: metrics.reporter.influxdb.consistency, ANY
2019-12-22 19:09:12,819 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: restart-strategy, failure-rate
2019-12-22 19:09:12,819 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.metrics.num-live-versions, true
2019-12-22 19:09:12,819 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: env.java.opts.taskmanager, -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:&lt;LOG_DIR&gt;/taskmanager-gc.log  -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=5M
2019-12-22 19:09:12,819 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.backend.rocksdb.writebuffer.size, 134217728
2019-12-22 19:09:12,819 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 4096m
2019-12-22 19:09:12,819 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: state.checkpoints.dir, hdfs://slothTest/user/sloth/sloth-fs-checkpoints/meta/1_7
2019-12-22 19:09:12,820 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: akka.client.timeout, 600 s
2019-12-22 19:09:12,846 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2019-12-22 19:09:12,851 INFO  org.apache.flink.runtime.clusterframework.BootstrapTools      - Setting directories for temporary files to: /mnt/dfs/0/yarn/local/usercache/sloth/appcache/application_1576548114502_0152,/mnt/dfs/1/yarn/local/usercache/sloth/appcache/application_1576548114502_0152,/home/sloth/hadoop/yarn/local/usercache/sloth/appcache/application_1576548114502_0152
2019-12-22 19:09:12,866 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting YarnJobClusterEntrypoint.
2019-12-22 19:09:12,867 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
2019-12-22 19:09:12,967 INFO  org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user set to sloth (auth:SIMPLE)
2019-12-22 19:09:12,982 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
2019-12-22 19:09:13,446 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at sloth-test1.dg.163.org:0
2019-12-22 19:09:14,207 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2019-12-22 19:09:14,239 INFO  akka.remote.Remoting                                          - Starting remoting
2019-12-22 19:09:14,425 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@sloth-test1.dg.163.org:16847]
2019-12-22 19:09:14,590 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@sloth-test1.dg.163.org:16847
2019-12-22 19:09:14,599 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'high-availability.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'
2019-12-22 19:09:14,603 INFO  com.filesystem.plugin.MyFileSystemFactory                     -  trying to get hadoopEnv, hadoopPath = /conf/hadoop_conf
2019-12-22 19:09:14,877 INFO  org.apache.hadoop.security.UserGroupInformation               - Login successful for user mammut_qa/dev@BDMS.163.COM using keytab file /tmp/zyf_test/mammut_qa.keytab
2019-12-22 19:09:15,123 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Shutting YarnJobClusterEntrypoint down with application status FAILED. Diagnostics java.io.IOException: Could not create FileSystem for highly available storage (high-availability.storageDir)
	at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:119)
	at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:92)
	at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:120)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:292)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:257)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:202)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501)
	at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:102)
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: slothTest
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2691)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:420)
	at com.filesystem.plugin.MyFileSystemKerberosMammutFactory3.getFileSystem(MyFileSystemKerberosMammutFactory3.java:63)
	at com.filesystem.plugin.MyFileSystemKerberosMammutFactory3.create(MyFileSystemKerberosMammutFactory3.java:38)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:438)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:359)
	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
	at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:116)
	... 13 more
Caused by: java.net.UnknownHostException: slothTest
	... 30 more
.
2019-12-22 19:09:15,130 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopping Akka RPC service.
2019-12-22 19:09:15,142 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
2019-12-22 19:09:15,144 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
2019-12-22 19:09:15,172 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
2019-12-22 19:09:15,207 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopped Akka RPC service.
2019-12-22 19:09:15,208 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Could not start cluster entrypoint YarnJobClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnJobClusterEntrypoint.
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501)
	at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:102)
Caused by: java.io.IOException: Could not create FileSystem for highly available storage (high-availability.storageDir)
	at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:119)
	at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:92)
	at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:120)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:292)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:257)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:202)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163)
	... 2 more
Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: slothTest
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:678)
	at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2691)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:420)
	at com.filesystem.plugin.MyFileSystemKerberosMammutFactory3.getFileSystem(MyFileSystemKerberosMammutFactory3.java:63)
	at com.filesystem.plugin.MyFileSystemKerberosMammutFactory3.create(MyFileSystemKerberosMammutFactory3.java:38)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:438)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:359)
	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
	at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:116)
	... 13 more
Caused by: java.net.UnknownHostException: slothTest
	... 30 more</pre>
    <div>
        <span>
            <br>
        </span>
    </div>
    <div id="ntes-pcmac-signature" style="font-family:'Helvetica','Microsoft Yahei', '微软雅黑'">
      
    <div style="font-size:14px; padding: 0;  margin:0;line-height: 14px;">
        <div style="border-bottom:1px solid #e6e6e6;display:inline-block;">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=ouywl&amp;uid=ouywl%40139.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsma8dc7719018ba2517da7111b3db5a170.jpg&amp;items=%5B%22ouywl%40139.com%22%5D" style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px 0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
            <table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
                <tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
                    <tr>
                            <td width="38" style="padding:0; box-sizing: border-box; width: 38px;">
                                <img width="38" height="38" style="vertical-align:middle; width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/sma8dc7719018ba2517da7111b3db5a170.jpg">
                            </td>
                            <td style="padding: 0 0 0 10px; color: #31353b;">
                                <div style="font-size: 16px;font-weight:bold; width:100%; white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">ouywl</div>
                            </td>
                    </tr>
                        <tr width="100%" style="font-size: 14px !important; width: 100%;">
                            <td colspan="2" style="padding:10px 0 0 0; font-size:14px !important; width: 100%;">
                                    <div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">ouywl@139.com</div>
                            </td>
                        </tr>
                </tbody>
            </table>
        </a>
        </div>
    </div>
    <div style="font-size:12px;color:#b5b9bd;margin-top:8px;line-height: 18px;">
        <span>签名由</span>
        <a style="text-decoration: none;color:#4196ff;padding:0px;" href="https://mail.163.com/dashi/dlpro.html?from=mail81">网易邮箱大师</a>
        <span>定制</span>
    </div>
 </div>
    <br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:''Helvetica','Microsoft Yahei', '微软雅黑'';">
    <div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="mail-date">12/19/2019 11:06</span>,<a class="mail-to" style="text-decoration:none;color:#2a83f2;" href="mailto:ouywl@139.com">ouywl&lt;ouywl@139.com&gt;</a> wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family: ''Helvetica','Microsoft Yahei', '微软雅黑'';">

<style>
    font{
        line-height: 1.6;
    }
    ul,ol{
        padding-left: 20px;
        list-style-position: inside;
    }
</style>
<div style="font-family:Helvetica,Helvetica,微软雅黑, 宋体; line-height:1.6;">
    <div></div><div>
    <div>
        <span>Hi Piotr Nowojski,</span></div><div><span>&nbsp; &nbsp;I have move my filesystem plugin to FLINK_HOME/pulgins in flink 1.9.1. The&nbsp;jobmanage don’t start up ,and It load the filesystem plugin in my owner plugin jar . and the log is :</span></div><div><span>&nbsp;&nbsp;“</span><span style="color: rgb(34, 34, 34); orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">2019-12-19 10:58:32,394 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'high-availability.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'</span></div><pre style="margin-top: 0px; margin-bottom: 0px; border: 0px; font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);"><font color="#222222">2019-12-19 10:58:32,398 INFO </font><b><font color="#e14141" class=" classDarkfont"> com.filesystem.plugin.FileSystemFactoryEnhance </font></b><font color="#222222">               -  trying to get hadoopEnv, hadoopPath = /conf/hadoop_conf
2019-12-19 10:58:32,434 WARN  org.apache.hadoop.conf.Configuration                          - /tmp/mammut-core-site.xml:an attempt to override final parameter: fs.defaultFS;  Ignoring.
2019-12-19 10:58:32,436 WARN  org.apache.hadoop.conf.Configuration                          - /tmp/mammut-hdfs-site.xml:an attempt to override final parameter: dfs.datanode.data.dir;  Ignoring.
2019-12-19 10:58:32,436 WARN  org.apache.hadoop.conf.Configuration                          - /tmp/mammut-hdfs-site.xml:an attempt to override final parameter: dfs.datanode.failed.volumes.tolerated;  Ignoring.
2019-12-19 10:58:32,436 WARN  org.apache.hadoop.conf.Configuration                          - /tmp/mammut-hdfs-site.xml:an attempt to override final parameter: dfs.namenode.name.dir;  Ignoring.
2019-12-19 10:58:32,878 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Shutting YarnJobClusterEntrypoint down with application status FAILED. Diagnostics java.io.IOException: Could not create FileSystem for highly available storage (high-availability.storageDir)
	at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:119)
	at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:92)
	at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:120)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:292)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:257)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:202)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:163)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501)&nbsp;</font></pre><div><span style="color: rgb(34, 34, 34); orphans: 2; widows: 2; background-color: rgb(255, 255, 255);">&nbsp;at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:102)</span><span>"</span></div>
    <div>
        <span>
            <br>
        </span>
    </div>
    <div id="ntes-pcmac-signature" style="font-family:'Helvetica','Microsoft Yahei', '微软雅黑'">
      
    <div style="font-size:14px; padding: 0;  margin:0;line-height: 14px;">
        <div style="border-bottom:1px solid #e6e6e6;display:inline-block;">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=ouywl&amp;uid=ouywl%40139.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsma8dc7719018ba2517da7111b3db5a170.jpg&amp;items=%5B%22ouywl%40139.com%22%5D" style="display:block;background:#fff; max-width: 400px; _width: 400px;padding:15px 0 10px 0;text-decoration: none; outline:none;-webkit-tap-highlight-color:transparent;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
            <table cellpadding="0" style="width: 100%; max-width: 100%; table-layout: fixed; border-collapse: collapse;color: #9b9ea1;font-size: 14px;line-height:1.3;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
                <tbody style="font-family: 'PingFang SC', 'Hiragino Sans GB','WenQuanYi Micro Hei', 'Microsoft Yahei', '微软雅黑', verdana !important; word-wrap:break-word; word-break:break-all;-webkit-text-size-adjust:none !important;text-size-adjust:none !important;">
                    <tr>
                            <td width="38" style="padding:0; box-sizing: border-box; width: 38px;">
                                <img width="38" height="38" style="vertical-align:middle; width: 38px; height: 38px; border-radius:50%;" src="https://mail-online.nosdn.127.net/sma8dc7719018ba2517da7111b3db5a170.jpg">
                            </td>
                            <td style="padding: 0 0 0 10px; color: #31353b;">
                                <div style="font-size: 16px;font-weight:bold; width:100%; white-space: nowrap; overflow:hidden;text-overflow: ellipsis;">ouywl</div>
                            </td>
                    </tr>
                        <tr width="100%" style="font-size: 14px !important; width: 100%;">
                            <td colspan="2" style="padding: 10px 0px 0px; width: 100%; font-size: 14px !important; word-break: break-all;">
                                    <div style="width: 100%;font-size: 14px !important;word-wrap:break-word;word-break:break-all;">ouywl@139.com</div></td></tr></tbody></table></a></div></div></div>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:''Helvetica','Microsoft Yahei', '微软雅黑'';">
    <div style="font-size:12px;line-height:1.6;word-break:break-all;margin-left:10px;margin-right:10px">On <span class="mail-date">12/19/2019 00:01</span>,<a class="mail-to" style="text-decoration:none;color:#2a83f2;" href="mailto:piotr@ververica.com">Piotr Nowojski&lt;piotr@ververica.com&gt;</a> wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family: ''Helvetica','Microsoft Yahei', '微软雅黑'';">
Hi,<div class=""><br class=""></div><div class="">As Yang Wang pointed out, you should use the new plugins mechanism.</div><div class=""><br class=""></div><div class="">If it doesn’t work, first make sure that you are shipping/distributing the plugins jars correctly - the correct plugins directory structure both on the client machine. Next make sure that the cluster has the same correct setup. This is especially true for the standalone/cluster execution modes. For yarn, mesos, docker the plugins dir should be shipped to the cluster by Flink itself, however Plugins support in yarn is currently semi broken [1]. This is already fixed, but waiting to be released in 1.9.2 and 1.10.</div><div class=""><br class=""></div><div class="">If it still doesn’t work, look for TaskManager logs what plugins/file systems are being loaded during the startup. If none, that's the problem.</div><div class=""><br class=""></div><div class="">Piotrek</div><div class=""><br class=""></div><div class="">[1]&nbsp;<a href="https://issues.apache.org/jira/browse/FLINK-14382" class="">https://issues.apache.org/jira/browse/FLINK-14382</a><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 18 Dec 2019, at 12:40, Yang Wang &lt;<a href="mailto:danrtsey.wy@gmail.com" class="">danrtsey.wy@gmail.com</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">You could have a try the new plugin&nbsp;mechanism.<div class="">Create a new directory named "myhdfs" under $FLINK_HOME/plugins, and then put your filesystem related jars in it.</div><div class="">Different plugins will be loaded by separate classloader to avoid conflict.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Best,</div><div class="">Yang</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">vino yang &lt;<a href="mailto:yanghua1127@gmail.com" class="">yanghua1127@gmail.com</a>&gt; 于2019年12月18日周三 下午6:46写道:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi ouywl,<div class=""><br class=""></div><div class=""><pre style="white-space:pre-wrap" class=""><b class=""><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">&gt;&gt;    </span><i style="font-family:Menlo;font-size:9pt" class="">Thread.currentThread().getContextClassLoader()</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span></b></pre></div><div class="">What does this statement mean in your program?</div><div class=""><br class=""></div><div class="">In addition, can you share your implementation of the customized file system plugin and the related exception?</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Vino</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">ouywl &lt;<a href="mailto:ouywl@139.com" target="_blank" class="">ouywl@139.com</a>&gt; 于2019年12月18日周三 下午4:59写道:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">

    

<div class="">

<div style="font-family:Helvetica,Helvetica,微软雅黑,宋体;line-height:1.6" class="">
    <div class=""></div><div class="">
    <div class="">
        <span class="">Hi all,</span></div><div class=""><span class="">&nbsp; &nbsp;&nbsp;</span>We have implemented a filesystem plugin for sink data to hdfs1, and the yarn for flink running is used hdfs2. So when the job running, the jobmanager use the conf of hdfs1 to create filesystem, the filesystem plugin &nbsp;is conflict with flink component.&nbsp;</div><div class="">&nbsp; &nbsp; We implemeted step:</div><div class="">&nbsp; &nbsp; &nbsp; 1. &nbsp;‘<span style="font-family:Menlo;font-size:9pt" class="">FileSystemEnhance</span>’ is implement from “<span style="font-family:Menlo;font-size:9pt" class="">FileSystem</span>”</div><div class="">&nbsp; &nbsp; &nbsp; 2. &nbsp;‘<span style="font-family:Menlo;font-size:9pt" class="">FileSystemFactoryEnhance</span>’ is implement from “<span style="font-family:Menlo;font-size:9pt" class="">FileSystemFactory</span>”,add kerberos auth in ”<span style="font-family:Menlo;font-size:12px" class="">FileSystemFactoryEnhance</span>"&nbsp;</div><div class="">&nbsp; &nbsp; &nbsp; 3.&nbsp;<span style="color:rgb(51,51,51);font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif;font-variant-ligatures:normal;background-color:rgb(255,255,255)" class="">Add a service entry. Create a file&nbsp;</span><code style="box-sizing: border-box; font-family: Menlo, &quot;Lucida Console&quot;, monospace; font-size: 12.6px; padding: 1px; background-color: rgb(255, 255, 255); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; font-variant-ligatures: normal;" class="">META-INF/services/org.apache.flink.core.fs.FileSystemFactory</code><span style="color:rgb(51,51,51);font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif;font-variant-ligatures:normal;background-color:rgb(255,255,255)" class="">&nbsp;which contains the class name of “</span><span style="font-family:Menlo;font-size:12px" class="">&nbsp;FileSystemFactoryEnhance.class</span><font color="#333333" face="Helvetica Neue, Helvetica, Arial, sans-serif" class=""><span class="">”</span></font></div><div class=""><br class=""></div><div class=""><font color="#333333" face="Helvetica Neue, Helvetica, Arial, sans-serif" class=""><span style="background-color:rgb(255,255,255)" class="">And &nbsp;the job mainclass is :</span></font></div><div class=""><font color="#333333" face="Helvetica Neue, Helvetica, Arial, sans-serif" class=""><span style="background-color:rgb(255,255,255)" class="">&nbsp; &nbsp;“&nbsp;</span></font><b class=""><i class=""><span style="font-family:Menlo;font-size:9pt" class="">public static void </span><span style="font-family:Menlo;font-size:9pt" class="">main</span><span style="font-family:Menlo;font-size:9pt" class="">(String[] args) </span><span style="font-family:Menlo;font-size:9pt" class="">throws </span><span style="font-family:Menlo;font-size:9pt" class="">Exception{</span></i></b></div><pre class=""><div style="text-align:left;font-family:Menlo;font-size:9pt" class=""><b style="font-size:9pt" class=""><i class="">    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();</i></b></div><b class=""><div style="text-align:left;font-family:Menlo;font-size:9pt" class=""><b style="font-size:9pt" class=""><i class="">    env.enableCheckpointing(60*1000);</i></b></div><div style="text-align:left;font-family:Menlo;font-size:9pt;font-style:italic" class=""><b style="font-size:9pt" class=""><i class="">    env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);</i></b></div><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-size:9pt" class=""><i class="">    env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);</i></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-size:9pt" class=""><i class="">    env.getConfig().enableSysoutLogging();</i></b></div></span><div style="text-align:left" class=""><font face="Menlo" class=""><span style="font-size:12px" class=""><i class=""><br class=""></i></span></font></div><div style="text-align:left" class=""><font face="Menlo" class=""><span style="font-size:12px" class=""><i class=""><br class=""></i></span></font></div><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">Properties props = </i><span style="font-family:Menlo;font-size:9pt" class="">new </span><i style="font-family:Menlo;font-size:9pt" class="">Properties()</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"bootstrap.servers"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><span style="font-family:Menlo;font-size:9pt" class="">SERVERS</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"<a href="http://group.id/" target="_blank" class="">group.id</a>"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><span style="font-family:Menlo;font-size:9pt" class="">GROUPID</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"enable.auto.commit"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><span style="font-family:Menlo;font-size:9pt" class="">"true"</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><span style="font-family:Menlo;font-size:9pt" class="">// props.put("<a href="http://auto.commit.interval.ms/" target="_blank" class="">auto.commit.interval.ms</a>", "1000");</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"<a href="http://session.timeout.ms/" target="_blank" class="">session.timeout.ms</a>"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><span style="font-family:Menlo;font-size:9pt" class="">"30000"</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"auto.offset.reset"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><span style="font-family:Menlo;font-size:9pt" class="">"latest"</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"key.deserializer"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><i style="font-family:Menlo;font-size:9pt" class="">org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.StringDeserializer.</i><span style="font-family:Menlo;font-size:9pt" class="">class</span><i style="font-family:Menlo;font-size:9pt" class="">.getName())</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">props.put(</i><span style="font-family:Menlo;font-size:9pt" class="">"value.deserializer"</span><span style="font-family:Menlo;font-size:9pt" class="">, </span><i style="font-family:Menlo;font-size:9pt" class="">StringDeserializer.</i><span style="font-family:Menlo;font-size:9pt" class="">class</span><i style="font-family:Menlo;font-size:9pt" class="">.getName())</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">FlinkKafkaConsumer010 consumer011 = </i><span style="font-family:Menlo;font-size:9pt" class="">new </span><i style="font-family:Menlo;font-size:9pt" class="">FlinkKafkaConsumer010&lt;String&gt;(</i><span style="font-family:Menlo;font-size:9pt" class="">"zyf_test_2"</span><span style="font-family:Menlo;font-size:9pt" class="">, new </span><i style="font-family:Menlo;font-size:9pt" class="">SimpleStringSchema()</i><span style="font-family:Menlo;font-size:9pt" class="">, </span><i style="font-family:Menlo;font-size:9pt" class="">props)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">DataStream&lt;String&gt; source = env.addSource(consumer011).setParallelism(</i><span style="font-family:Menlo;font-size:9pt" class="">1</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><div style="text-align:left" class=""><font face="Menlo" class=""><span style="font-size:12px" class=""><i class=""><br class=""></i></span></font></div><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">source.print()</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">Thread.currentThread().getContextClassLoader()</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><div style="text-align:left" class=""><font face="Menlo" class=""><span style="font-size:12px" class=""><i class=""><br class=""></i></span></font></div><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">StreamingFileSink sink = StreamingFileSink</i></b></div></span><i style="font-family:Menlo;font-size:9pt" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><i style="font-family:Menlo;font-size:9pt" class="">            .forRowFormat(</i><span style="font-family:Menlo;font-size:9pt" class="">new </span><i style="font-family:Menlo;font-size:9pt" class="">Path(</i><span style="font-family:Menlo;font-size:9pt" class="">"<a href="hdfs://bdms-test/user/sloth/zyf" class="">hdfs://bdms-test/user/sloth/zyf</a>"</span><i style="font-family:Menlo;font-size:9pt" class="">)</i><span style="font-family:Menlo;font-size:9pt" class="">, new </span><i style="font-family:Menlo;font-size:9pt" class="">SimpleStringEncoder&lt;&gt;(</i><span style="font-family:Menlo;font-size:9pt" class="">"UTF-8"</span><i style="font-family:Menlo;font-size:9pt" class="">))</i></b></div></i><i style="font-family:Menlo;font-size:9pt" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><i style="font-family:Menlo;font-size:9pt" class="">            .build()</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></i><div style="text-align:left" class=""><font face="Menlo" class=""><span style="font-size:12px" class=""><i class=""><br class=""></i></span></font></div><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">source.addSink(sink)</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><div style="text-align:left" class=""><font face="Menlo" class=""><span style="font-size:12px" class=""><i class=""><br class=""></i></span></font></div><span style="font-family:Menlo;font-size:9pt;font-style:italic" class=""><div style="text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><span style="font-family:Menlo;font-size:9pt" class="">    </span><i style="font-family:Menlo;font-size:9pt" class="">env.execute()</i><span style="font-family:Menlo;font-size:9pt" class="">;</span></b></div></span><div style="font-family:Menlo;font-size:9pt;font-style:italic;text-align:left" class=""><b style="font-family:Helvetica;font-size:14px" class=""><i style="font-family:Menlo;font-size:9pt" class="">}</i></b><span style="background-color:rgb(255,255,255);font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif;font-size:14px" class="">”</span></div><div style="font-family:Menlo;font-size:9pt;font-style:italic;text-align:left" class=""><span style="background-color:rgb(255,255,255);font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif;font-size:14px" class=""><br class=""></span></div><div style="text-align:left" class=""><span style="font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif;font-size:14px;font-style:italic;background-color:rgb(255,255,255)" class="">And start the job, the jobmanager filesystem is error, the log means the jobmananger use “</span><span style="font-family:Menlo;font-size:9pt;font-style:italic;white-space:normal" class="">FileSystemFactoryEnhance</span><span style="background-color:rgb(255,255,255)" class=""><font face="Helvetica Neue, Helvetica, Arial, sans-serif" class=""><i class="">”</i></font><font face="Helvetica Neue, Helvetica, Arial, sans-serif" style="font-family:Menlo;font-size:9pt" class=""><span style="font-size:14px" class=""><i class="">&nbsp;filesystem and confict.</i></span></font></span></div><div style="text-align:left" class=""><span style="background-color:rgb(255,255,255)" class=""><font face="Helvetica Neue, Helvetica, Arial, sans-serif" style="font-family:Menlo;font-size:9pt" class=""><span style="font-size:14px" class=""><i class=""><br class=""></i></span></font></span></div><div class=""><font face="Menlo" class=""><i class="">As the url </i></font><a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/#pluggable-file-systems" style="font-family:Helvetica" target="_blank" class="">https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/#pluggable-file-systems</a> how to avoid use “<b style="font-family:Menlo;font-size:9pt" class="">Thread.<span style="font-style:italic" class="">currentThread</span>().getContextClassLoader()</b><span style="font-family:Helvetica" class="">"</span></div></b></pre>
    <div class="">
        <span class="">
            <br class="">
        </span>
    </div>
    <div id="gmail-m_-348939560936265204gmail-m_2696436437561713147ntes-pcmac-signature" style="font-family:Helvetica,&quot;Microsoft Yahei&quot;,微软雅黑" class="">
      
    <div style="font-size:14px;padding:0px;margin:0px;line-height:14px" class="">
        <div style="border-bottom:1px solid rgb(230,230,230);display:inline-block" class="">
                    <a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&amp;name=ouywl&amp;uid=ouywl%40139.com&amp;iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsma8dc7719018ba2517da7111b3db5a170.jpg&amp;items=%5B%22ouywl%40139.com%22%5D" style="display:block;background:rgb(255,255,255);max-width:400px;padding:15px 0px 10px;text-decoration:none;outline:none" target="_blank" class="">
            <table cellpadding="0" style="width:100%;max-width:100%;table-layout:fixed;border-collapse:collapse;color:rgb(155,158,161);font-size:14px;line-height:1.3" class="">
                <tbody style="word-break:break-all;font-family:&quot;PingFang SC&quot;,&quot;Hiragino Sans GB&quot;,&quot;WenQuanYi Micro Hei&quot;,&quot;Microsoft Yahei&quot;,微软雅黑,verdana" class="">
                    <tr class="">
                            <td width="38" style="padding:0px;box-sizing:border-box;width:38px">
                                <img width="38" height="38" style="vertical-align: middle; width: 38px; height: 38px; border-radius: 50%;" src="https://mail-online.nosdn.127.net/sma8dc7719018ba2517da7111b3db5a170.jpg" class="">
                            </td>
                            <td style="padding:0px 0px 0px 10px;color:rgb(49,53,59)">
                                <div style="font-size:16px;font-weight:bold;width:100%;white-space:nowrap;overflow:hidden;text-overflow:ellipsis" class="">ouywl</div>
                            </td>
                    </tr>
                        <tr width="100%" style="width:100%;font-size:14px" class="">
                            <td colspan="2" style="padding:10px 0px 0px;width:100%;word-break:break-all;font-size:14px">
                                    <div style="width:100%;word-break:break-all;font-size:14px" class="">ouywl@139.com</div></td></tr></tbody></table></a></div></div>
 </div>
    <br class="">
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div>
</blockquote><!--�-->
</div></blockquote><!--�-->
</div>
</body>
</html>
Mime
View raw message