Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 11 Jul 2016 18:01:11 +0000 (UTC)
From: "Yongjun Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12984537.1467155930000.81628.1468260071202@Atlassian.JIRA>
In-Reply-To: <JIRA.12984537.1467155930000@Atlassian.JIRA>
References: <JIRA.12984537.1467155930000@Atlassian.JIRA> <JIRA.12984537.1467155930948@arcas>
Subject: [jira] [Commented] (HDFS-10587) Incorrect offset/length calculation
 in pipeline recovery causes block corruption
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 11 Jul 2016 18:01:13 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10587?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1537=
1294#comment-15371294 ]=20

Yongjun Zhang commented on HDFS-10587:
--------------------------------------

HI [~jojochuang],

I think it'd be nice to work out a unit test that demonstrates the block co=
rruption, for example, to create a block with visibleLength X, and a replic=
a with data X+delta written to disk,  then use the involved code to copy th=
e replica to a different one, to see if  the corruption happens. If so, the=
n we can see my above proposed change can address the issue.=20

Of course, we still need to understand better the "chunk end enforcement" m=
entioned in my earlier comment.=20
=20
What do you think?

Thanks.


> Incorrect offset/length calculation in pipeline recovery causes block cor=
ruption
> -------------------------------------------------------------------------=
-------
>
>                 Key: HDFS-10587
>                 URL: https://issues.apache.org/jira/browse/HDFS-10587
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>
> We found incorrect offset and length calculation in pipeline recovery may=
 cause block corruption and results in missing blocks under a very unfortun=
ate scenario.=20
> (1) A client established pipeline and started writing data to the pipelin=
e.
> (2) One of the data node in the pipeline restarted, closing the socket, a=
nd some written data were unacknowledged.
> (3) Client replaced the failed data node with a new one, initiating block=
 transfer to copy existing data in the block to the new datanode.
> (4) The block is transferred to the new node. Crucially, the entire block=
, including the unacknowledged data, was transferred.
> (5) The last chunk (512 bytes) was not a full chunk, but the destination =
still reserved the whole chunk in its buffer, and wrote the entire buffer t=
o disk, therefore some written data is garbage.
> (6) When the transfer was done, the destination data node converted the r=
eplica from temporary to rbw, which made its visible length as the length o=
f bytes on disk. That is to say, it thought whatever was transferred was ac=
knowledged. However, the visible length of the replica is different (round =
up to the next multiple of 512) than the source of transfer.
> (7) Client then truncated the block in the attempt to remove unacknowledg=
ed data. However, because the visible length is equivalent of the bytes on =
disk, it did not truncate unacknowledged data.
> (8) When new data was appended to the destination, it skipped the bytes a=
lready on disk. Therefore, whatever was written as garbage was not replaced=
.
> (9) the volume scanner detected corrupt replica, but due to HDFS-10512, i=
t wouldn=E2=80=99t tell NameNode to mark the replica as corrupt, so the cli=
ent continued to form a pipeline using the corrupt replica.
> (10) Finally the DN that had the only healthy replica was restarted. Name=
Node then update the pipeline to only contain the corrupt replica.
> (11) Client continue to write to the corrupt replica, because neither cli=
ent nor the data node itself knows the replica is corrupt. When the restart=
ed datanodes comes back, their replica are stale, despite they are not corr=
upt. Therefore, none of the replica is good and up to date.
> The sequence of events was reconstructed based on DataNode/NameNode log a=
nd my understanding of code.
> Incidentally, we have observed the same sequence of events on two indepen=
dent clusters.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org