Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E58C8DABC for ; Sat, 27 Oct 2012 17:35:30 +0000 (UTC) Received: (qmail 16282 invoked by uid 500); 27 Oct 2012 17:35:25 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 16197 invoked by uid 500); 27 Oct 2012 17:35:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 16189 invoked by uid 99); 27 Oct 2012 17:35:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Oct 2012 17:35:25 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.210.48] (HELO mail-da0-f48.google.com) (209.85.210.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Oct 2012 17:35:20 +0000 Received: by mail-da0-f48.google.com with SMTP id z8so1793821dad.35 for ; Sat, 27 Oct 2012 10:34:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:x-originating-ip:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :x-gm-message-state; bh=JiJT6kMsy/3qNAyqYowzAeGRiOdlyiVYnhZRUWw2EFU=; b=pNYmi28R5GBlLJBMrpA1C3dkjR03vybrYtkfWpKIKTHVNYmskeP266BajqHLZFtpv+ YBy/CIcOuGh+TprOspzglF3GGGpnBZ/7yTi2d56229nSakCZLtunnUity+p5qTDmTqYK U0l8UA8BMZVrjUPui4Vb0cHhlIEbOP9ha9mAm6VrikcDpEJ6dNpx/PR8RPpebIW42zvE B2hxKHl68fEe0XovZstqgoK+jmzLIuej1+kJXK0GNm7uyxYMkglBuO9s/Uk8ePNahzbT ne/mGRiGn5Azib63913jcxIhwX+JowJpSS/GfkrUHm7kDRxGOpWon/7n+WG7hVm+Izy/ oagg== MIME-Version: 1.0 Received: by 10.68.237.167 with SMTP id vd7mr70205130pbc.161.1351359298697; Sat, 27 Oct 2012 10:34:58 -0700 (PDT) Sender: balaji@balajin.net Received: by 10.68.6.71 with HTTP; Sat, 27 Oct 2012 10:34:58 -0700 (PDT) X-Originating-IP: [50.131.42.219] In-Reply-To: References: Date: Sat, 27 Oct 2012 10:34:58 -0700 X-Google-Sender-Auth: 0bciISvAEOa0PvWsa60y8ciLKj0 Message-ID: Subject: Re: HDFS HA IO Fencing From: =?UTF-8?B?QmFsYWppIE5hcmF5YW5hbiAo4K6q4K6+4K6y4K6+4K6c4K6/IOCuqOCuvuCusOCuvuCur+Cuow==?= =?UTF-8?B?4K6p4K+NKQ==?= To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=047d7b339025f6792804cd0ddac2 X-Gm-Message-State: ALoCoQnrP8wnDANI3gQHvl4dIzUTq7z0MHb5DaAVdUHDiHBYilVcbpKrSRWAyAVlSkr6mM/Bs3eN X-Virus-Checked: Checked by ClamAV on apache.org --047d7b339025f6792804cd0ddac2 Content-Type: text/plain; charset=UTF-8 If you use NSFv4 you should be able to use locks and when a machine dies / fails to renew the lease, the other machine can take over. On Friday, October 26, 2012, Todd Lipcon wrote: > NFS Locks typically last forever if you disconnect abruptly. So they are > not sufficient -- your standby wouldn't be able to take over without manual > intervention to remove the lock. > > If you want to build an unreliable system that might corrupt your data, > you could set up 'shell(/bin/true)' as a second fencing method. But, it's > really a bad idea. There are failure scenarios which could cause split > brain if you do this, and you'd very likely lose data. > > -Todd > > On Fri, Oct 26, 2012 at 1:59 AM, lei liu > > wrote: > >> We are using NFS for Shared storage, Can we use linux nfslcok service to >> implement IO Fencing ? >> >> >> 2012/10/26 Steve Loughran > 'cvml', 'stevel@hortonworks.com');>> >> >>> >>> >>> On 25 October 2012 14:08, Todd Lipcon >>> > wrote: >>> >>>> Hi Liu, >>>> >>>> Locks are not sufficient, because there is no way to enforce a lock in >>>> a distributed system without unbounded blocking. What you might be >>>> referring to is a lease, but leases are still problematic unless you can >>>> put bounds on the speed with which clocks progress on different machines, >>>> _and_ have strict guarantees on the way each node's scheduler works. With >>>> Linux and Java, the latter is tough. >>>> >>>> >>> on any OS running in any virtual environment, including EC2, time is >>> entirely unpredictable, just to make things worse. >>> >>> >>> On a single machine you can use file locking as the OS will know that >>> the process is dead and closes the file; other programs can attempt to open >>> the same file with exclusive locking -and, by getting the right failures, >>> know that something else has the file, hence the other process is live. >>> Shared NFS storage you need to mount with softlock set precisely to stop >>> file locks lasting until some lease has expired, because the on-host >>> liveness probes detect failure faster and want to react to it. >>> >>> >>> -Steve >>> >> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera > -- Thanks -balaji -- http://balajin.net/blog/ http://flic.kr/balajijegan --047d7b339025f6792804cd0ddac2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable If you use NSFv4 you should be able to use locks and when a machine dies / = fails to renew the lease, the other machine can take over.

On Frida= y, October 26, 2012, Todd Lipcon wrote:
NFS Locks typically last forever if you disconnect abruptly. So they are no= t sufficient -- your standby wouldn't be able to take over without manu= al intervention to remove the lock.

If you want to build= an unreliable system that might corrupt your data, you could set up 's= hell(/bin/true)' as a second fencing method. But, it's really a bad= idea. There are failure scenarios which could cause split brain if you do = this, and you'd very likely lose data.

-Todd

On Fri, Oc= t 26, 2012 at 1:59 AM, lei liu <l= iulei412@gmail.com> wrote:
We are using NFS for Shared storage,=C2=A0 = Can we use linux nfslcok service to implement IO Fencing ?


2012/10/26 Steve Loughran = <stevel@hortonworks.com>


On 25 October 2012 14:08, Todd Lipc= on <todd@cloudera.com> wrote:
Hi Liu,

Locks are not sufficient, because there is no wa= y to enforce a lock in a distributed system without unbounded blocking. Wha= t you might be referring to is a lease, but leases are still problematic un= less you can put bounds on the speed with which clocks progress on differen= t machines, _and_ have strict guarantees on the way each node's schedul= er works. With Linux and Java, the latter is tough.


on any OS running in = any virtual environment, including EC2, time is entirely unpredictable, jus= t to make things worse.=C2=A0


On a = single machine you can use file locking as the OS will know that the proces= s is dead and closes the file; other programs can attempt to open the same = file with exclusive locking -and, by getting the right failures, know that = something else has the file, hence the other process is live. Shared NFS st= orage you need to mount with softlock set precisely to stop file locks last= ing until some lease has expired, because the on-host liveness probes detec= t failure faster and want to react to it.


-Steve




--
= Todd Lipcon
Software Engineer, Cloudera


--
Thanks
-balaji

--
http://balajin.net/blog/
http://flic.kr/balajijegan

--047d7b339025f6792804cd0ddac2--