Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Subject: Re: libhdfs force close hdfsFile
MIME-Version: 1.0
From: Chris Nauroth <cnauroth@hortonworks.com>
To: Ken Huang <dnionhkx@gmail.com>, "user@hadoop.apache.org"
	<user@hadoop.apache.org>
Thread-Topic: libhdfs force close hdfsFile
Thread-Index: AQHRcDf+VkPpfeXP3k6fo27QEnkHep8+z6qA
Date: Fri, 26 Feb 2016 20:55:42 +0000
Message-ID: <D2F5F994.3C116%cnauroth@hortonworks.com>
References: 
 <CAHKU=1kSvZSXXHLYpXbYTkSUU1hc6DrmAkdCZKrbLe9jVmAQcw@mail.gmail.com>
In-Reply-To: 
 <CAHKU=1kSvZSXXHLYpXbYTkSUU1hc6DrmAkdCZKrbLe9jVmAQcw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [50.248.208.113]
x-source-routing-agent: Processed
Content-Type: multipart/alternative;
	boundary="_000_D2F5F9943C116cnaurothhortonworkscom_"

--_000_D2F5F9943C116cnaurothhortonworkscom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello Ken,

The closest thing to what you're requesting is in the Java API, there is th=
e slightly dodgy, semi-private, we-hope-only-HBase-calls-it method Distribu=
tedFileSystem#recoverLease.  This is capable of telling the NameNode to rec=
over the lease (and ultimately close the file if necessary) based on any sp=
ecified path.  This method is not exposed through libhdfs though, and just =
so it's clear, I wouldn't recommend using it even if it was.

When I hear questions like this, it's often because an application is writi=
ng to a file at a certain path and there is a desire for recoverability if =
the application terminates prematurely, such as due to a server crash.  Use=
rs would like another process to be able to take over right away and start =
writing to the file again, but the NameNode won't allow this until after ex=
piration of the old client's lease.  Is this the use case you had in mind?

If so, then a pattern that can work well is for the application to create a=
nd write to a unique temporary file name instead of the final destination p=
ath.  Then, after writing all data, the application renames the temporary f=
ile to the desired final destination.  Since the leases are tracked on the =
file paths being written, the old client's lease on its temporary file won'=
t block the new client from writing to a different temporary file.

--Chris Nauroth

From: Ken Huang <dnionhkx@gmail.com<mailto:dnionhkx@gmail.com>>
Date: Thursday, February 25, 2016 at 5:49 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.ap=
ache.org<mailto:user@hadoop.apache.org>>
Subject: libhdfs force close hdfsFile

Hi,

Does anyone know how to close a hdfsFile while the connection between hdfsC=
lient and NameNode is lost ?

Thanks
Ken Huang

--_000_D2F5F9943C116cnaurothhortonworkscom_
Content-Type: text/html; charset="us-ascii"
Content-ID: <9D8088B166C1D24BB0DC2DF78100AE14@exch080.serverpod.net>
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami=
ly: Calibri, sans-serif;">
<div>
<div>
<div>Hello Ken,</div>
<div><br>
</div>
<div>The closest thing to what you're requesting is in the Java API, there =
is the slightly dodgy, semi-private, we-hope-only-HBase-calls-it method Dis=
tributedFileSystem#recoverLease. &nbsp;This is capable of telling the NameN=
ode to recover the lease (and ultimately
 close the file if necessary) based on any specified path. &nbsp;This metho=
d is not exposed through libhdfs though, and just so it's clear, I wouldn't=
 recommend using it even if it was.</div>
<div><br>
</div>
<div>When I hear questions like this, it's often because an application is =
writing to a file at a certain path and there is a desire for recoverabilit=
y if the application terminates prematurely, such as due to a server crash.=
 &nbsp;Users would like another process
 to be able to take over right away and start writing to the file again, bu=
t the NameNode won't allow this until after expiration of the old client's =
lease. &nbsp;Is this the use case you had in mind?</div>
<div><br>
</div>
<div>If so, then a pattern that can work well is for the application to cre=
ate and write to a unique temporary file name instead of the final destinat=
ion path. &nbsp;Then, after writing all data, the application renames the t=
emporary file to the desired final destination.
 &nbsp;Since the leases are tracked on the file paths being written, the ol=
d client's lease on its temporary file won't block the new client from writ=
ing to a different temporary file.</div>
<div><br>
</div>
<div><font class=3D"Apple-style-span" color=3D"#000000"><font class=3D"Appl=
e-style-span" face=3D"Calibri">--Chris Nauroth</font></font></div>
</div>
</div>
<div><br>
</div>
<span id=3D"OLK_SRC_BODY_SECTION">
<div style=3D"font-family:Calibri; font-size:11pt; text-align:left; color:b=
lack; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM:=
 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid;=
 BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style=3D"font-weight:bold">From: </span>Ken Huang &lt;<a href=3D"mail=
to:dnionhkx@gmail.com">dnionhkx@gmail.com</a>&gt;<br>
<span style=3D"font-weight:bold">Date: </span>Thursday, February 25, 2016 a=
t 5:49 PM<br>
<span style=3D"font-weight:bold">To: </span>&quot;<a href=3D"mailto:user@ha=
doop.apache.org">user@hadoop.apache.org</a>&quot; &lt;<a href=3D"mailto:use=
r@hadoop.apache.org">user@hadoop.apache.org</a>&gt;<br>
<span style=3D"font-weight:bold">Subject: </span>libhdfs force close hdfsFi=
le<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir=3D"ltr">Hi,
<div><br>
</div>
<div>Does anyone know how to close a hdfsFile while the connection between =
hdfsClient and NameNode is lost ?</div>
<div><br>
</div>
<div>Thanks</div>
<div>Ken Huang</div>
</div>
</div>
</div>
</span>
</body>
</html>

--_000_D2F5F9943C116cnaurothhortonworkscom_--