hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDDS-722) ozone datanodes failed to start on few nodes
Date Sun, 04 Nov 2018 07:16:00 GMT

     [ https://issues.apache.org/jira/browse/HDDS-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mukul Kumar Singh resolved HDDS-722.
------------------------------------
    Resolution: Duplicate

This issue is fixed with HDDS-762. Duping it

> ozone datanodes failed to start on few nodes
> --------------------------------------------
>
>                 Key: HDDS-722
>                 URL: https://issues.apache.org/jira/browse/HDDS-722
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Datanode
>    Affects Versions: 0.3.0
>            Reporter: Nilotpal Nandi
>            Priority: Critical
>         Attachments: all-node-ozone-logs-1540356965.tar.gz
>
>
> steps taken :
> ------------------
>  # put few keys using ozonefs.
>  # stopped all services of the cluster.
>  # started om and scm.
>  # After sometime , started datanodes.
> All datanodes failed to start . Out of 12 datanodes, 4 datanodes failed to start.
>  
> Here is the datanode log snippet :
> ------------------------------------------------
>  
> {noformat}
> 2018-10-24 04:49:30,594 ERROR org.apache.ratis.server.impl.StateMachineUpdater: Terminating
with exit status 2: StateMachineUpdater-9524f4e2-9031-4852-ab7c-11c2da3460db: the StateMachineUpdater
hits Throwable
> org.apache.ratis.server.storage.RaftLogIOException: java.io.IOException: Premature EOF
from inputStream
>  at org.apache.ratis.server.storage.LogSegment.loadCache(LogSegment.java:299)
>  at org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:192)
>  at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:142)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Premature EOF from inputStream
>  at org.apache.ratis.util.IOUtils.readFully(IOUtils.java:100)
>  at org.apache.ratis.server.storage.LogReader.decodeEntry(LogReader.java:250)
>  at org.apache.ratis.server.storage.LogReader.readEntry(LogReader.java:155)
>  at org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:128)
>  at org.apache.ratis.server.storage.LogSegment.readSegmentFile(LogSegment.java:110)
>  at org.apache.ratis.server.storage.LogSegment.access$400(LogSegment.java:43)
>  at org.apache.ratis.server.storage.LogSegment$LogEntryLoader.load(LogSegment.java:167)
>  at org.apache.ratis.server.storage.LogSegment$LogEntryLoader.load(LogSegment.java:161)
>  at org.apache.ratis.server.storage.LogSegment.loadCache(LogSegment.java:295)
>  ... 3 more
> 2018-10-24 04:49:30,598 INFO org.apache.hadoop.ozone.HddsDatanodeService: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down HddsDatanodeService at ctr-e138-1518143905142-541661-01-000003.hwx.site/172.27.57.0
> ************************************************************/
> 2018-10-24 04:49:30,598 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread Interrupted
waiting to refresh disk information: sleep interrupted
>  
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message