hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Maheshwari (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDDS-610) On restart of SCM it fails to register DataNodes
Date Tue, 09 Oct 2018 23:49:00 GMT
Namit Maheshwari created HDDS-610:
-------------------------------------

             Summary: On restart of SCM it fails to register DataNodes
                 Key: HDDS-610
                 URL: https://issues.apache.org/jira/browse/HDDS-610
             Project: Hadoop Distributed Data Store
          Issue Type: Bug
            Reporter: Namit Maheshwari


{code:java}
2018-10-09 23:34:11,105 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG: host = ctr-e138-1518143905142-510793-01-000004.hwx.site/172.27.79.197
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.3.0-SNAPSHOT
STARTUP_MSG: classpath = /tmp/ozone-0.3.0-SNAPSHOT/etc/hadoop:/tmp/ozone-0.3.0-SNAPSHOT/share/hadoop/common/*:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-core-2.9.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/protobuf-java-2.5.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/netty-3.10.5.Final.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-beanutils-1.9.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/okio-1.6.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-core-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-security-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/bcpkix-jdk15on-1.54.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-client-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsr305-3.0.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/leveldbjni-all-1.8.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/j2objc-annotations-1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jctools-core-2.1.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-common-0.8.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-common-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jzlib-1.1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-value-1.6.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-util-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-server-framework-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-servlet-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-xdr-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-client-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-core-asl-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/lz4-1.3.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-net-3.6.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-netty-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/log4j-api-2.11.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/netty-all-4.0.52.Final.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hamcrest-all-1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/error_prone_annotations-2.2.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-config-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-service-1.0-rc4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/metrics-core-3.2.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-pkix-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/xz-1.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/rocksdbjni-5.14.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-container-service-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-auth-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsr311-api-1.1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-annotations-2.9.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/javax.servlet-api-3.1.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-server-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/zookeeper-3.4.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/avro-1.7.7.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-client-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-xc-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/disruptor-3.4.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/guava-11.0.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/curator-framework-2.12.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-databind-2.9.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/dnsjava-2.1.7.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-common-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-configuration2-2.1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/json-smart-2.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/token-provider-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/log4j-1.2.17.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-daemon-1.0.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-util-ajax-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/picocli-3.5.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-grpc-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-lang3-3.7.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-compress-1.4.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/accessors-smart-1.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/bcprov-jdk15on-1.54.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jaxb-impl-2.2.3-1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/re2j-1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/curator-recipes-2.12.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-math3-3.1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-xml-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/httpclient-4.5.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/asm-5.0.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-common-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/curator-client-2.12.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/audience-annotations-0.5.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-util-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-server-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jaxb-api-2.2.11.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsp-api-2.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-server-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsch-0.1.54.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/compress-lzf-1.0.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-util-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/netty-tcnative-2.0.8.Final-osx-x86_64.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/paranamer-2.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/log4j-core-2.11.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-http-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-codec-1.11.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-pool2-2.6.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/gson-2.2.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jcip-annotations-1.0-1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-admin-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-io-2.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/snakeyaml-1.16.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-core-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/javapoet-1.10.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-webapp-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jboss-marshalling-1.4.11.Final.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/aalto-xml-1.0.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdfs-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-cli-1.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/woodstox-core-5.0.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/snappy-java-1.0.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-proto-shaded-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-asn1-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/stax2-api-3.1.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-collections-3.2.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jettison-1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-text-1.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-servlet-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdfs-client-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-json-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/slf4j-api-1.7.25.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/okhttp-2.7.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-logging-1.1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-io-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-identity-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/animal-sniffer-annotations-1.16.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-common-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-server-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-value-annotations-1.6.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-crypto-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/httpcore-4.4.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/web:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-server-scm-0.3.0-SNAPSHOT.jar:/opt/byteman.jar
STARTUP_MSG: build = git@github.com:apache/hadoop.git -r 347ea385817766a5c418017009728cd8b9959776;
compiled by 'nmaheshwari' on 2018-10-09T00:14Z
STARTUP_MSG: java = 1.8.0_112
************************************************************/
2018-10-09 23:34:11,119 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: registered
UNIX signal handlers for [TERM, HUP, INT]
2018-10-09 23:34:11,376 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Entering startup
chill mode.
2018-10-09 23:34:12,106 INFO org.apache.hadoop.hdds.scm.server.SCMChillModeManager: SCM exiting
chill mode.
2018-10-09 23:34:12,107 WARN org.apache.hadoop.hdds.server.events.EventQueue: No event handler
registered for event TypedEvent{payloadType=Boolean, name='Boolean'}
2018-10-09 23:34:12,107 WARN org.apache.hadoop.hdds.server.events.EventQueue: No event handler
registered for event TypedEvent{payloadType=Boolean, name='Boolean'}
2018-10-09 23:34:12,202 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2018-10-09 23:34:12,420 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler,
ipcBackoff: false.
2018-10-09 23:34:12,448 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
9861
2018-10-09 23:34:12,508 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler,
ipcBackoff: false.
2018-10-09 23:34:12,509 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
9863
2018-10-09 23:34:12,539 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler,
ipcBackoff: false.
2018-10-09 23:34:12,541 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
9860
2018-10-09 23:34:12,580 INFO org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for scm at:
http://0.0.0.0:9876
2018-10-09 23:34:12,626 INFO org.eclipse.jetty.util.log: Logging initialized @2524ms
2018-10-09 23:34:12,800 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2018-10-09 23:34:12,824 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.scm
is not defined
2018-10-09 23:34:12,840 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2018-10-09 23:34:12,844 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context scm
2018-10-09 23:34:12,845 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2018-10-09 23:34:12,845 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2018-10-09 23:34:12,960 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: StorageContainerLocationProtocol
RPC server is listening at /0.0.0.0:9860
2018-10-09 23:34:13,113 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties
from hadoop-metrics2.properties
2018-10-09 23:34:13,134 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
Metric snapshot period at 10 second(s).
2018-10-09 23:34:13,134 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: StorageContainerManager
metrics system started
2018-10-09 23:34:13,189 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: RPC
server for Client is listening at /0.0.0.0:9860
2018-10-09 23:34:13,190 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-10-09 23:34:13,190 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9860: starting
2018-10-09 23:34:13,193 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: ScmBlockLocationProtocol
RPC server is listening at /0.0.0.0:9863
2018-10-09 23:34:13,193 INFO org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer: RPC
server for Block Protocol is listening at /0.0.0.0:9863
2018-10-09 23:34:13,193 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-10-09 23:34:13,194 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9863: starting
2018-10-09 23:34:13,197 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: ScmDatanodeProtocl
RPC server is listening at /0.0.0.0:9861
2018-10-09 23:34:13,197 INFO org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer:
RPC server for DataNodes is listening at /0.0.0.0:9861
2018-10-09 23:34:13,198 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-10-09 23:34:13,199 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9861: starting
2018-10-09 23:34:13,208 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 9876
2018-10-09 23:34:13,209 INFO org.eclipse.jetty.server.Server: jetty-9.3.19.v20170502
2018-10-09 23:34:13,264 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2484f433{/logs,file:///tmp/ozone-0.3.0-SNAPSHOT/logs/,AVAILABLE}
2018-10-09 23:34:13,265 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5f59185e{/static,jar:file:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-server-scm-0.3.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2018-10-09 23:34:13,379 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@24528a25{/,file:///tmp/jetty-0.0.0.0-9876-scm-_-any-4060966483262888676.dir/webapp/,AVAILABLE}{/scm}
2018-10-09 23:34:13,387 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@46292372{HTTP/1.1,[http/1.1]}{0.0.0.0:9876}
2018-10-09 23:34:13,387 INFO org.eclipse.jetty.server.Server: Started @3284ms
2018-10-09 23:34:13,388 INFO org.apache.hadoop.hdds.server.BaseHttpServer: HTTP server of
SCM is listening at http://0.0.0.0:9876
2018-10-09 23:34:13,440 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat
from unregistered datanode 636d2ebe-b1c0-4885-b07d-b6624d583258{ip: 172.27.23.193, host: ctr-e138-1518143905142-510793-01-000006.hwx.site}
2018-10-09 23:34:13,455 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler:
Error on processing container report from datanode 636d2ebe-b1c0-4885-b07d-b6624d583258{ip:
172.27.23.193, host: ctr-e138-1518143905142-510793-01-000006.hwx.site}
org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-10-09 23:34:31,325 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat
from unregistered datanode 06edf838-e6b1-4dea-9557-a008007cfb1b{ip: 172.27.17.133, host: ctr-e138-1518143905142-510793-01-000010.hwx.site}
2018-10-09 23:34:40,241 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat
from unregistered datanode 5086b8e0-1aa4-4c50-b4db-859f9a9519f7{ip: 172.27.87.64, host: ctr-e138-1518143905142-510793-01-000005.hwx.site}
2018-10-09 23:34:40,242 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler:
Error on processing container report from datanode 5086b8e0-1aa4-4c50-b4db-859f9a9519f7{ip:
172.27.87.64, host: ctr-e138-1518143905142-510793-01-000005.hwx.site}
org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-10-09 23:34:40,888 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with
ID: 636d2ebe-b1c0-4885-b07d-b6624d583258 Registered.
2018-10-09 23:34:43,789 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat
from unregistered datanode 1baa3f7c-d85a-4ab5-b005-d2a740233e32{ip: 172.27.52.130, host: ctr-e138-1518143905142-510793-01-000007.hwx.site}
2018-10-09 23:34:43,790 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler:
Error on processing container report from datanode 1baa3f7c-d85a-4ab5-b005-d2a740233e32{ip:
172.27.52.130, host: ctr-e138-1518143905142-510793-01-000007.hwx.site}
org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-10-09 23:34:49,416 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat
from unregistered datanode e9c4a12f-db11-4e19-98fc-d1ec7b9d1aeb{ip: 172.27.68.128, host: ctr-e138-1518143905142-510793-01-000008.hwx.site}
2018-10-09 23:35:01,324 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with
ID: 06edf838-e6b1-4dea-9557-a008007cfb1b Registered.
2018-10-09 23:35:03,358 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat
from unregistered datanode cf462de6-6709-48ef-b822-f9270e13bc6f{ip: 172.27.16.145, host: ctr-e138-1518143905142-510793-01-000011.hwx.site}
2018-10-09 23:35:03,359 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler:
Error on processing container report from datanode cf462de6-6709-48ef-b822-f9270e13bc6f{ip:
172.27.16.145, host: ctr-e138-1518143905142-510793-01-000011.hwx.site}
org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Once the DN's are restarted they are registered fine
{code:java}
2018-10-09 23:35:10,239 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with
ID: 5086b8e0-1aa4-4c50-b4db-859f9a9519f7 Registered.
2018-10-09 23:35:13,786 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with
ID: 1baa3f7c-d85a-4ab5-b005-d2a740233e32 Registered.
2018-10-09 23:35:19,407 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with
ID: e9c4a12f-db11-4e19-98fc-d1ec7b9d1aeb Registered.
2018-10-09 23:35:33,357 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with
ID: cf462de6-6709-48ef-b822-f9270e13bc6f Registered.{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message