Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 14 Jul 2016 22:06:20 +0000 (UTC)
From: "Yongjun Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12984537.1467155930000.35301.1468533980676@Atlassian.JIRA>
In-Reply-To: <JIRA.12984537.1467155930000@Atlassian.JIRA>
References: <JIRA.12984537.1467155930000@Atlassian.JIRA> <JIRA.12984537.1467155930948@arcas>
Subject: [jira] [Commented] (HDFS-10587) Incorrect offset/length calculation
 in pipeline recovery causes block corruption
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 14 Jul 2016 22:06:23 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10587?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1537=
8483#comment-15378483 ]=20

Yongjun Zhang commented on HDFS-10587:
--------------------------------------

I think this looks why we had new data reaching the new DN after the init b=
lock transfer:  after adding the new DN to the pipeline, doing the block tr=
ansfer to this new DN, the client resumed writing data. Then in the process=
, corruption is detected again, thus repeating the pipeline recovery proces=
s. Even though from client side point of view, it keeps getting the followi=
ng exception

{code}
INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 10.1.1.1:1110
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockO=
utputStream(DFSOutputStream.java:1472)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelin=
eForAppendOrRecovery(DFSOutputStream.java:1293)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatan=
odeError(DFSOutputStream.java:1016)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutpu=
tStream.java:560)
{code}

Wei-Chiu and I discussed, and we think here is a more complete picture:

* 1. pipeline going on DN1 -> DN2 -> DN3
* 2. trouble at DN3, it's gone
* 3. pipeline recovery, new DN DN4 added
* 4. block transfer from DN1 to DN4, DN4's data is now a multiple of chunks=
.
* 5. DataStreamer resumed writing data to DN1 -> DN4 -> DN3 (this is where =
new data gets in), the first chunk DN4 got is corrupt for some reason that =
we are searching for
* 6. DN3 detects corruption, quit; while new data has been written to DN1 a=
nd DN4
* 7. goes back to step 3, new pipeline recovery starts
DN1 ->DN4 -> DN5
DN1 -> DN4 -> DN6
......

At a corner case, Step 3 could be replaced with "DN3 restarted", in which c=
ase, another block transfer would happen, and may cause corruption.

Since DN1's visibleLength in step 4 is not a multiple of chunks, this fact =
might be somehow related to the corruption in step 5.

> Incorrect offset/length calculation in pipeline recovery causes block cor=
ruption
> -------------------------------------------------------------------------=
-------
>
>                 Key: HDFS-10587
>                 URL: https://issues.apache.org/jira/browse/HDFS-10587
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>         Attachments: HDFS-10587.001.patch
>
>
> We found incorrect offset and length calculation in pipeline recovery may=
 cause block corruption and results in missing blocks under a very unfortun=
ate scenario.=20
> (1) A client established pipeline and started writing data to the pipelin=
e.
> (2) One of the data node in the pipeline restarted, closing the socket, a=
nd some written data were unacknowledged.
> (3) Client replaced the failed data node with a new one, initiating block=
 transfer to copy existing data in the block to the new datanode.
> (4) The block is transferred to the new node. Crucially, the entire block=
, including the unacknowledged data, was transferred.
> (5) The last chunk (512 bytes) was not a full chunk, but the destination =
still reserved the whole chunk in its buffer, and wrote the entire buffer t=
o disk, therefore some written data is garbage.
> (6) When the transfer was done, the destination data node converted the r=
eplica from temporary to rbw, which made its visible length as the length o=
f bytes on disk. That is to say, it thought whatever was transferred was ac=
knowledged. However, the visible length of the replica is different (round =
up to the next multiple of 512) than the source of transfer. [1]
> (7) Client then truncated the block in the attempt to remove unacknowledg=
ed data. However, because the visible length is equivalent of the bytes on =
disk, it did not truncate unacknowledged data.
> (8) When new data was appended to the destination, it skipped the bytes a=
lready on disk. Therefore, whatever was written as garbage was not replaced=
.
> (9) the volume scanner detected corrupt replica, but due to HDFS-10512, i=
t wouldn=E2=80=99t tell NameNode to mark the replica as corrupt, so the cli=
ent continued to form a pipeline using the corrupt replica.
> (10) Finally the DN that had the only healthy replica was restarted. Name=
Node then update the pipeline to only contain the corrupt replica.
> (11) Client continue to write to the corrupt replica, because neither cli=
ent nor the data node itself knows the replica is corrupt. When the restart=
ed datanodes comes back, their replica are stale, despite they are not corr=
upt. Therefore, none of the replica is good and up to date.
> The sequence of events was reconstructed based on DataNode/NameNode log a=
nd my understanding of code.
> Incidentally, we have observed the same sequence of events on two indepen=
dent clusters.
> [1]
> The sender has the replica as follows:
> 2016-04-15 22:03:05,066 INFO org.apache.hadoop.hdfs.server.datanode.fsdat=
aset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_1556997324_110=
0153495099, RBW
>   getNumBytes()     =3D 41381376
>   getBytesOnDisk()  =3D 41381376
>   getVisibleLength()=3D 41186444
>   getVolume()       =3D /hadoop-i/data/current
>   getBlockFile()    =3D /hadoop-i/data/current/BP-1043567091-10.1.1.1-134=
3682168507/current/rbw/blk_1556997324
>   bytesAcked=3D41186444
>   bytesOnDisk=3D41381376
> while the receiver has the replica as follows:
> 2016-04-15 22:03:05,068 INFO org.apache.hadoop.hdfs.server.datanode.fsdat=
aset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_1556997324_110=
0153495099, RBW
>   getNumBytes()     =3D 41186816
>   getBytesOnDisk()  =3D 41186816
>   getVisibleLength()=3D 41186816
>   getVolume()       =3D /hadoop-g/data/current
>   getBlockFile()    =3D /hadoop-g/data/current/BP-1043567091-10.1.1.1-134=
3682168507/current/rbw/blk_1556997324
>   bytesAcked=3D41186816
>   bytesOnDisk=3D41186816


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org