Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 22 Feb 2017 18:43:44 +0000 (UTC)
From: "Manoj Govindassamy (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13044707.1487650951000.34034.1487789024502@Atlassian.JIRA>
In-Reply-To: <JIRA.13044707.1487650951000@Atlassian.JIRA>
References: <JIRA.13044707.1487650951000@Atlassian.JIRA> <JIRA.13044707.1487650951460@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-11435) NameNode should track open for
 write files lengths more frequent than on newer block allocations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 22 Feb 2017 18:43:50 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11435?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1587=
8952#comment-15878952 ]=20

Manoj Govindassamy commented on HDFS-11435:
-------------------------------------------

Thanks [~linyiqun] for the reference to HDFS-11194. Will take a look.

[~raviprak],=20
I get your points. On normal circumstances, we will not be needing the _nea=
r_ realtime lengths of OPEN_FOR_WRITE files. If at all needed, as Jing poin=
ted out, there are already provisions for clients to reach out to DataNodes=
 directly to find the the latest lengths for a being written file. The inte=
ntion here is not check the progress of a slow writer. I believe the curren=
t LeaseManager soft/hard lease limits are good enough for tackling very slo=
w writer issues.

The intention of this jira is to close a gap in HDFS Snapshots w.r.t open f=
iles. As many other metadata only operations, HDFS Snapshots are NameNode o=
nly operations and there by the file lengths captured are as good as what i=
s available in NN at the Snapshot time. So, for the files that are open and=
 being written, NN lags the latest file lengths by as much as a block size =
and there by these open files that are captured in Snapshots have incorrect=
 lengths. The current behavior of HDFS Snapshots is to let these open files=
 in the Snapshots also grow/shrink just like the original file, and finaliz=
e it only after its open file is closed. Thus HDFS Snapshots are not truly =
_read-only_ w.r.t open files. HDFS -11402 attempts to close this gap and ma=
ke HDFS Snapshots truly read-only by freezing these open files in Snapshots=
 via meta data copy. To make the design proposed in above jira more reliabl=
e, we need NN getting to know lengths of open files more frequently than th=
e current model. More discussions on this are available in HDFS-11402. Plea=
se let me know if you need more details.=20


> NameNode should track open for write files lengths more frequent than on =
newer block allocations
> -------------------------------------------------------------------------=
-----------------------
>
>                 Key: HDFS-11435
>                 URL: https://issues.apache.org/jira/browse/HDFS-11435
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Manoj Govindassamy
>            Assignee: Manoj Govindassamy
>
> *Problem:*
> Currently the length of an open for write / Under construction file is up=
dated on the NameNode only when=20
> # Block boundary: On block boundaries and upon allocation of new Block, N=
ameNode gets to know the file growth and the file length catches up
> # hsync(SyncFlag.UPDATE_LENGTH): Upon Client apps invoking a hsync on the=
 write stream with a special flag, DataNodes send an incremental block repo=
rt with the latest file length which NameNode uses it to update its meta da=
ta.
> # First hflush() on the new Block: Upon Client apps doing first time hflu=
sh() on an every new Block, DataNodes notifies NameNode about the latest fi=
le length.
> # Output stream close: Forces DataNodes update NameNode about the file le=
ngth after data persistence and proper acknowledgements in the pipeline.
> So, lengths for open for write files are usually a lot less than the leng=
th seen by the DN/client. Highly preferred to have NameNode not lagging in =
file lengths by order of Block size for under construction files and to hav=
e more frequent, scalable update mechanism for these open file lengths.=20


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org