hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7820) Client Write fails after rolling upgrade rollback with "<block_id> already exist in finalized state"
Date Thu, 12 Mar 2015 06:07:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358144#comment-14358144
] 

Vinayakumar B commented on HDFS-7820:
-------------------------------------

bq. Increasing the epoch bits to 5 or 6 will permit successive rollbacks in a short time interval
while avoiding collisions. 
I agree for one version upgrade and rollback. 
But my point is, whatever bits changed will remain forever for the entire lifetime of the
cluster. We cannot change it back, otherwise blockIds will be having the lesser value than
previous. For 6 bits this gives 2^6 rollbacks at max.
Another issue is, we cannot persist this, since at this time NN will be at that startup phase.
So in this case, if Namenode restarts again without writing any single block, then problem
can occur again.

Am I missing something?

bq. This behavior was added before my time but I think deferring deletions speeds up cluster
startup e.g. after upgrades. DN startup time is already an issue in large clusters.
I agree for the upgrade. But for the rollback, entire cluster has to be restarted anyway.

Another option is to write new blocks after to some new directory inside volume itself after
rolling upgrade starts, similar to trash, ex: upgrade . And on finalize of upgrade move to
finalized directory. And on Rollback just delete them.

> Client Write fails after rolling upgrade rollback with "<block_id> already exist
in finalized state"
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7820
>                 URL: https://issues.apache.org/jira/browse/HDFS-7820
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: J.Andreina
>            Assignee: J.Andreina
>         Attachments: HDFS-7820.1.patch
>
>
> Steps to Reproduce:
> ===================
> Step 1:  Prepare rolling upgrade using "hdfs dfsadmin -rollingUpgrade prepare"
> Step 2:  Shutdown SNN and NN
> Step 3:  Start NN with the "hdfs namenode -rollingUpgrade started" option.
> Step 4:  Executed "hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade"
and restarted Datanode
> Step 5:  Write 3 files to hdfs ( block id assigned are : blk_1073741831_1007, blk_1073741832_1008,blk_1073741833_1009
)
> Step 6:  Shutdown both NN and DN
> Step 7:  Start NNs with the "hdfs namenode -rollingUpgrade rollback" option.
>          Start DNs with the "-rollback" option.
> Step 8:  Write 2 files to hdfs.
> Issue:
> =======
> Client write failed with below exception
> {noformat}
> 2015-02-23 16:00:12,896 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving
BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 src: /XXXXXXXXXXX:48545 dest:
/XXXXXXXXXXX:50010
> 2015-02-23 16:00:12,897 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException:
Block BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741832_1008 already exists in state
FINALIZED and thus cannot be created.
> {noformat}
> Observations:
> =============
> 1. At Namenode side block invalidate is been sent only to 2 blocks.
> {noformat}
> 15/02/23 14:59:56 INFO BlockStateChange: BLOCK* InvalidateBlocks: add blk_1073741833_1009
to XXXXXXXXXXX:50010
> 15/02/23 14:59:56 INFO BlockStateChange: BLOCK* InvalidateBlocks: add blk_1073741831_1007
to XXXXXXXXXXX:50010
> {noformat}
> 2. fsck report does not show information on blk_1073741832_1008
> {noformat}
> FSCK started by Rex (auth:SIMPLE) from /XXXXXXXXXXX for path / at Mon Feb 23 16:17:57
CST 2015
> /File1:  Under replicated BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741825_1001.
Target Replicas is 3 but found 1 replica(s).
> /File11:  Under replicated BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741827_1003.
Target Replicas is 3 but found 1 replica(s).
> /File2:  Under replicated BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741826_1002.
Target Replicas is 3 but found 1 replica(s).
> /AfterRollback_2:  Under replicated BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741831_1007.
Target Replicas is 3 but found 1 replica(s).
> /Test1:  Under replicated BP-1837556285-XXXXXXXXXXX-1423130389269:blk_1073741828_1004.
Target Replicas is 3 but found 1 replica(s).
> Status: HEALTHY
>  Total size:    31620 B
>  Total dirs:    7
>  Total files:   6
>  Total symlinks:                0
>  Total blocks (validated):      5 (avg. block size 6324 B)
>  Minimally replicated blocks:   5 (100.0 %)
>  Over-replicated blocks:        0 (0.0 %)
>  Under-replicated blocks:       5 (100.0 %)
>  Mis-replicated blocks:         0 (0.0 %)
>  Default replication factor:    3
>  Average block replication:     1.0
>  Corrupt blocks:                0
>  Missing replicas:              10 (66.666664 %)
>  Number of data-nodes:          1
>  Number of racks:               1
> FSCK ended at Mon Feb 23 16:17:57 CST 2015 in 3 milliseconds
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message