hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10999) Use more generic "low redundancy" blocks instead of "under replicated" blocks
Date Tue, 18 Oct 2016 16:45:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585942#comment-15585942
] 

Wei-Chiu Chuang edited comment on HDFS-10999 at 10/18/16 4:45 PM:
------------------------------------------------------------------

Just FYI, the fsck output in Hadoop 3 is of the following format. It does separate replicated
blocks from erasure coded blocks. [~aw] how does this output look for you from an admin perspective?

{noformat}
FSCK started by weichiu (auth:SIMPLE) from /127.0.0.1 for path / at Tue Oct 18 09:43:23 PDT
2016
/ <dir>
/striped <dir>
/striped/missing 393216 bytes, 1 block(s):  MISSING 1 blocks of total size 393216 B
0. BP-406181414-172.16.1.88-1476808998435:blk_-9223372036854775792_1001 len=393216 MISSING!
Live_repl=5  [DatanodeInfoWithStorage[127.0.0.1:56489,DS-1722aa0e-1ade-4286-a09f-fca1999b5370,DISK],
DatanodeInfoWithStorage[127.0.0.1:56481,DS-fc14e048-de5e-4dbd-986d-78161d37bcb2,DISK], DatanodeInfoWithStorage[127.0.0.1:56485,DS-128523b5-086f-4b24-bb52-0426937debc3,DISK],
DatanodeInfoWithStorage[127.0.0.1:56493,DS-5dbac29b-1c82-40b2-b22b-7958915afd72,DISK], DatanodeInfoWithStorage[127.0.0.1:56497,DS-a19f46a3-6357-42b7-84f3-9ac3ddf05ba1,DISK]]


Status: CORRUPT
 Number of data-nodes:	5
 Number of racks:		1
 Total dirs:			2
 Total symlinks:		0

Replicated Blocks:
 Total size:	0 B
 Total files:	0
 Total blocks (validated):	0
 Minimally replicated blocks:	0
 Over-replicated blocks:	0
 Under-replicated blocks:	0
 Mis-replicated blocks:		0
 Default replication factor:	3
 Average block replication:	0.0
 Missing blocks:		0
 Corrupt blocks:		0
 Missing replicas:		0

Erasure Coded Block Groups:
 Total size:	393216 B
 Total files:	1
 Total block groups (validated):	1 (avg. block group size 393216 B)
  ********************************
  UNRECOVERABLE BLOCK GROUPS:	1 (100.0 %)
  CORRUPT FILES:	1
  MISSING BLOCK GROUPS:	1
  MISSING SIZE:		393216 B
  ********************************
 Minimally erasure-coded block groups:	0 (0.0 %)
 Over-erasure-coded block groups:	0 (0.0 %)
 Under-erasure-coded block groups:	0 (0.0 %)
 Unsatisfactory placement block groups:	0 (0.0 %)
 Default ecPolicy:		RS-DEFAULT-6-3-64k
 Average block group size:	5.0
 Missing block groups:		1
 Corrupt block groups:		0
 Missing internal blocks:	0 (0.0 %)
FSCK ended at Tue Oct 18 09:43:23 PDT 2016 in 2 milliseconds


The filesystem under path '/' is CORRUPT

{noformat}


was (Author: jojochuang):
Just FYI, the fsck output in Hadoop 3 is of the following format. It does separate replicated
blocks from erasure coded blocks. [~aw] how does this output look for you from an admin perspective?

{noformat}
FSCK started by weichiu (auth:SIMPLE) from /127.0.0.1 for path / at Tue Oct 18 09:37:00 PDT
2016

/striped/corrupted: CORRUPT blockpool BP-921842435-172.16.1.88-1476808612846 block blk_-9223372036854775792

/striped/corrupted: CORRUPT 1 blocks of total size 393216 B.
Status: CORRUPT
 Number of data-nodes:	9
 Number of racks:		1
 Total dirs:			2
 Total symlinks:		0

Replicated Blocks:
 Total size:	0 B
 Total files:	0
 Total blocks (validated):	0
 Minimally replicated blocks:	0
 Over-replicated blocks:	0
 Under-replicated blocks:	0
 Mis-replicated blocks:		0
 Default replication factor:	3
 Average block replication:	0.0
 Missing blocks:		0
 Corrupt blocks:		0
 Missing replicas:		0

Erasure Coded Block Groups:
 Total size:	393216 B
 Total files:	1
 Total block groups (validated):	1 (avg. block group size 393216 B)
  ********************************
  UNRECOVERABLE BLOCK GROUPS:	1 (100.0 %)
  CORRUPT FILES:	1
  CORRUPT BLOCK GROUPS: 	1
  CORRUPT SIZE:		393216 B
  ********************************
 Minimally erasure-coded block groups:	0 (0.0 %)
 Over-erasure-coded block groups:	0 (0.0 %)
 Under-erasure-coded block groups:	0 (0.0 %)
 Unsatisfactory placement block groups:	0 (0.0 %)
 Default ecPolicy:		RS-DEFAULT-6-3-64k
 Average block group size:	5.0
 Missing block groups:		0
 Corrupt block groups:		1
 Missing internal blocks:	0 (0.0 %)
FSCK ended at Tue Oct 18 09:37:00 PDT 2016 in 2 milliseconds


The filesystem under path '/' is CORRUPT

{noformat}

> Use more generic "low redundancy" blocks instead of "under replicated" blocks
> -----------------------------------------------------------------------------
>
>                 Key: HDFS-10999
>                 URL: https://issues.apache.org/jira/browse/HDFS-10999
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Yuanbo Liu
>              Labels: supportability
>
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic term "low
redundancy" to the old-fashioned "under replicated". But this term is still being used in
messages in several places, such as web ui, dfsadmin and fsck. We should probably change them
to avoid confusion.
> File this jira to discuss it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message