Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of stevel@hortonworks.com
 designates 209.85.216.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADY20s6XkN_Cub6+QVOjxUMyf2WHkCLT18etp9vjh1HRq7ZjfA@mail.gmail.com>
References: 
 <CABbGW3wG0390=bmhSi1P=8J-ZSr373fEQk=YNe6idjFGiAEdqQ@mail.gmail.com>
	<CADY20s6XkN_Cub6+QVOjxUMyf2WHkCLT18etp9vjh1HRq7ZjfA@mail.gmail.com>
Date: Thu, 25 Oct 2012 19:23:03 +0100
Message-ID: 
 <CA+4kjVvJihNu0oQ42gv0XArFek7gQsoUQRz9ej+8ttiK7r9CSw@mail.gmail.com>
Subject: Re: HDFS HA IO Fencing
From: Steve Loughran <stevel@hortonworks.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf303640cf3a1e7704cce64bc4

--20cf303640cf3a1e7704cce64bc4
Content-Type: text/plain; charset=UTF-8

On 25 October 2012 14:08, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Liu,
>
> Locks are not sufficient, because there is no way to enforce a lock in a
> distributed system without unbounded blocking. What you might be referring
> to is a lease, but leases are still problematic unless you can put bounds
> on the speed with which clocks progress on different machines, _and_ have
> strict guarantees on the way each node's scheduler works. With Linux and
> Java, the latter is tough.
>
>
on any OS running in any virtual environment, including EC2, time is
entirely unpredictable, just to make things worse.


On a single machine you can use file locking as the OS will know that the
process is dead and closes the file; other programs can attempt to open the
same file with exclusive locking -and, by getting the right failures, know
that something else has the file, hence the other process is live. Shared
NFS storage you need to mount with softlock set precisely to stop file
locks lasting until some lease has expired, because the on-host liveness
probes detect failure faster and want to react to it.


-Steve

--20cf303640cf3a1e7704cce64bc4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<br><br><div class=3D"gmail_quote">On 25 October 2012 14:08, Todd Lipcon <s=
pan dir=3D"ltr">&lt;<a href=3D"mailto:todd@cloudera.com" target=3D"_blank">=
todd@cloudera.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote=
" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Liu,<div><br></div><div>Locks are not sufficient, because there is no wa=
y to enforce a lock in a distributed system without unbounded blocking. Wha=
t you might be referring to is a lease, but leases are still problematic un=
less you can put bounds on the speed with which clocks progress on differen=
t machines, _and_ have strict guarantees on the way each node&#39;s schedul=
er works. With Linux and Java, the latter is tough.</div>


<div><br></div></blockquote><div><br></div><div>on any OS running in any vi=
rtual environment, including EC2, time is entirely unpredictable, just to m=
ake things worse.=C2=A0</div><div><br></div><div><br></div><div>On a single=
 machine you can use file locking as the OS will know that the process is d=
ead and closes the file; other programs can attempt to open the same file w=
ith exclusive locking -and, by getting the right failures, know that someth=
ing else has the file, hence the other process is live. Shared NFS storage =
you need to mount with softlock set precisely to stop file locks lasting un=
til some lease has expired, because the on-host liveness probes detect fail=
ure faster and want to react to it.</div>
<div><br></div><div><br></div><div>-Steve</div></div>

--20cf303640cf3a1e7704cce64bc4--