Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 19 Jul 2016 07:02:20 +0000 (UTC)
From: "Yongjun Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12984537.1467155930000.69145.1468911740786@Atlassian.JIRA>
In-Reply-To: <JIRA.12984537.1467155930000@Atlassian.JIRA>
References: <JIRA.12984537.1467155930000@Atlassian.JIRA> <JIRA.12984537.1467155930948@arcas>
Subject: [jira] [Commented] (HDFS-10587) Incorrect offset/length calculation
 in pipeline recovery causes block corruption
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 19 Jul 2016 07:02:22 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10587?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1538=
3685#comment-15383685 ]=20

Yongjun Zhang commented on HDFS-10587:
--------------------------------------

Hi [~vinayrpet],

Thanks much again for creating the testcase! Upon further investigation, we=
 found that our branch initially had HDFS-4660, then it was reverted becaus=
e of an issue, which was later identified as HDFS-9220. We had a tool to qu=
ery jiras in a given branch, and it failed to tell us the jira was reverted=
, so we thought we already had HDFS-4660. And we kept thinking it's somethi=
ng else, though we saw the symptom here is really same as HDFS-4660.

I did an experiment by reverting HDFS-4660 etc, and found your test failed =
as expected. I created HDFS-10652 for adding this test as a unit test for H=
DFS-4660. if you don't mind, would you please assign it to yourself, craft =
it and add comments as needed, and attach new versions there?

Thanks a lot!


> Incorrect offset/length calculation in pipeline recovery causes block cor=
ruption
> -------------------------------------------------------------------------=
-------
>
>                 Key: HDFS-10587
>                 URL: https://issues.apache.org/jira/browse/HDFS-10587
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>         Attachments: HDFS-10587-test.patch, HDFS-10587.001.patch
>
>
> We found incorrect offset and length calculation in pipeline recovery may=
 cause block corruption and results in missing blocks under a very unfortun=
ate scenario.=20
> (1) A client established pipeline and started writing data to the pipelin=
e.
> (2) One of the data node in the pipeline restarted, closing the socket, a=
nd some written data were unacknowledged.
> (3) Client replaced the failed data node with a new one, initiating block=
 transfer to copy existing data in the block to the new datanode.
> (4) The block is transferred to the new node. Crucially, the entire block=
, including the unacknowledged data, was transferred.
> (5) The last chunk (512 bytes) was not a full chunk, but the destination =
still reserved the whole chunk in its buffer, and wrote the entire buffer t=
o disk, therefore some written data is garbage.
> (6) When the transfer was done, the destination data node converted the r=
eplica from temporary to rbw, which made its visible length as the length o=
f bytes on disk. That is to say, it thought whatever was transferred was ac=
knowledged. However, the visible length of the replica is different (round =
up to the next multiple of 512) than the source of transfer. [1]
> (7) Client then truncated the block in the attempt to remove unacknowledg=
ed data. However, because the visible length is equivalent of the bytes on =
disk, it did not truncate unacknowledged data.
> (8) When new data was appended to the destination, it skipped the bytes a=
lready on disk. Therefore, whatever was written as garbage was not replaced=
.
> (9) the volume scanner detected corrupt replica, but due to HDFS-10512, i=
t wouldn=E2=80=99t tell NameNode to mark the replica as corrupt, so the cli=
ent continued to form a pipeline using the corrupt replica.
> (10) Finally the DN that had the only healthy replica was restarted. Name=
Node then update the pipeline to only contain the corrupt replica.
> (11) Client continue to write to the corrupt replica, because neither cli=
ent nor the data node itself knows the replica is corrupt. When the restart=
ed datanodes comes back, their replica are stale, despite they are not corr=
upt. Therefore, none of the replica is good and up to date.
> The sequence of events was reconstructed based on DataNode/NameNode log a=
nd my understanding of code.
> Incidentally, we have observed the same sequence of events on two indepen=
dent clusters.
> [1]
> The sender has the replica as follows:
> 2016-04-15 22:03:05,066 INFO org.apache.hadoop.hdfs.server.datanode.fsdat=
aset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_1556997324_110=
0153495099, RBW
>   getNumBytes()     =3D 41381376
>   getBytesOnDisk()  =3D 41381376
>   getVisibleLength()=3D 41186444
>   getVolume()       =3D /hadoop-i/data/current
>   getBlockFile()    =3D /hadoop-i/data/current/BP-1043567091-10.1.1.1-134=
3682168507/current/rbw/blk_1556997324
>   bytesAcked=3D41186444
>   bytesOnDisk=3D41381376
> while the receiver has the replica as follows:
> 2016-04-15 22:03:05,068 INFO org.apache.hadoop.hdfs.server.datanode.fsdat=
aset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_1556997324_110=
0153495099, RBW
>   getNumBytes()     =3D 41186816
>   getBytesOnDisk()  =3D 41186816
>   getVisibleLength()=3D 41186816
>   getVolume()       =3D /hadoop-g/data/current
>   getBlockFile()    =3D /hadoop-g/data/current/BP-1043567091-10.1.1.1-134=
3682168507/current/rbw/blk_1556997324
>   bytesAcked=3D41186816
>   bytesOnDisk=3D41186816


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org