Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4D9A0ED96 for ; Wed, 16 Jan 2013 04:47:47 +0000 (UTC) Received: (qmail 47326 invoked by uid 500); 16 Jan 2013 04:47:42 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 46741 invoked by uid 500); 16 Jan 2013 04:47:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 46600 invoked by uid 99); 16 Jan 2013 04:47:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jan 2013 04:47:38 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rakeshr@huawei.com designates 119.145.14.65 as permitted sender) Received: from [119.145.14.65] (HELO szxga02-in.huawei.com) (119.145.14.65) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jan 2013 04:47:33 +0000 Received: from 172.24.2.119 (EHLO szxeml209-edg.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.3.4-GA FastPath queued) with ESMTP id AVS42918; Wed, 16 Jan 2013 12:47:11 +0800 (CST) Received: from SZXEML457-HUB.china.huawei.com (10.82.67.200) by szxeml209-edg.china.huawei.com (172.24.2.184) with Microsoft SMTP Server (TLS) id 14.1.323.3; Wed, 16 Jan 2013 12:47:09 +0800 Received: from SZXEML531-MBX.china.huawei.com ([fe80::61a8:2cb5:62f9:d4a4]) by szxeml457-hub.china.huawei.com ([10.82.67.200]) with mapi id 14.01.0323.003; Wed, 16 Jan 2013 12:47:07 +0800 From: Rakesh R To: "user@hadoop.apache.org" Subject: RE: hadoop namenode recovery Thread-Topic: hadoop namenode recovery Thread-Index: AQHN8r896T3oI1KOc0Km5peYlCp25JhJSEgAgAFfVICAACynAIAAjW6O Date: Wed, 16 Jan 2013 04:47:06 +0000 Message-ID: References: <50F6039D.5050201@comcast.net>, In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.96.96] Content-Type: multipart/alternative; boundary="_000_C2496325850AA74C92AAF83AA9662D2630E335B8szxeml531mbxchi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Virus-Checked: Checked by ClamAV on apache.org --_000_C2496325850AA74C92AAF83AA9662D2630E335B8szxeml531mbxchi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, I feel the most reliable approach is using NN-HA features with shared stora= ge. Here the idea is having two Namenodes. Both the Active, Standby(Seconda= ry) Namenodes will be pointing to the shared device and writes the editlogs= to it. When the Active crashes, Standby will take over and become Active a= nd continue serving the clients reliably without much interruptions. One of the possible approach is with BookKeeper as Shared storage device: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHigh= Availability.html#BookKeeper_as_a_Shared_storage_EXPERIMENTAL http://zookeeper.apache.org/doc/trunk/bookkeeperOverview.html Thanks, Rakesh ________________________________ From: Harsh J [harsh@cloudera.com] Sent: Wednesday, January 16, 2013 9:44 AM To: Subject: Re: hadoop namenode recovery The NFS mount is to be soft-mounted; so if the NFS goes down, the NN ejects= it out and continues with the local disk. If auto-restore is configured, i= t will re-add the NFS if its detected good again later. On Wed, Jan 16, 2013 at 7:04 AM, randy > wrote: What happens to the NN and/or performance if there's a problem with the NFS= server? Or the network? Thanks, randy On 01/14/2013 11:36 PM, Harsh J wrote: Its very rare to observe an NN crash due to a software bug in production. Most of the times its a hardware fault you should worry about. On 1.x, or any non-HA-carrying release, the best you can get to safeguard against a total loss is to have redundant disk volumes configured, one preferably over a dedicated remote NFS mount. This way the NN is recoverable after the node goes down, since you can retrieve a current copy from another machine (i.e. via the NFS mount) and set a new node up to replace the older NN and continue along. A load balancer will not work as the NN is not a simple webserver - it maintains state which you cannot sync. We wrote HA-HDFS features to address the very concern you have. If you want true, painless HA, branch-2 is your best bet at this point. An upcoming 2.0.3 release should include the QJM based HA features that is painless to setup and very reliable to use (over other options), and works with commodity level hardware. FWIW, we've (my team and I) been supporting several users and customers who're running the 2.x based HA in production and other types of environments and it has been greatly stable in our experience. There are also some folks in the community running 2.x based HDFS for HA/else. On Tue, Jan 15, 2013 at 6:55 AM, Panshul Whisper >> wrote: Hello, Is there a standard way to prevent the failure of Namenode crash in a Hadoop cluster? or what is the standard or best practice for overcoming the Single point failure problem of Hadoop. I am not ready to take chances on a production server with Hadoop 2.0 Alpha release, which claims to have solved the problem. Are there any other things I can do to either prevent the failure or recover from the failure in a very short time. Thanking You, -- Regards, Ouch Whisper 010101010101 -- Harsh J -- Harsh J --_000_C2496325850AA74C92AAF83AA9662D2630E335B8szxeml531mbxchi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi,

 

I feel the most reliable approach is using NN-HA features with shared st= orage. Here the idea is having two Namenodes. Both the Active, Standby(Seco= ndary) Namenodes will be pointing to the shared device and writes the editl= ogs to it. When the Active crashes, Standby will take over and become Active and continue serving the&nbs= p;clients reliably without much interruptions.

 

 

One of the possible approach is with BookKeeper as Shared storage device= :
h= ttp://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHighA= vailability.html#BookKeeper_as_a_Shared_storage_EXPERIMENTAL

 

http://zookeeper.apache.org/doc/trunk/bookkeeperOverview.html

 

Thanks,

Rakesh

From: Harsh J [harsh@cloudera.com]
Sent: Wednesday, January 16, 2013 9:44 AM
To: <user@hadoop.apache.org>
Subject: Re: hadoop namenode recovery

The NFS mount is to be soft-mounted; so if the NFS goes do= wn, the NN ejects it out and continues with the local disk. If auto-restore= is configured, it will re-add the NFS if its detected good again later.


On Wed, Jan 16, 2013 at 7:04 AM, randy <rand= ysch@comcast.net> wrote:
What happens to the NN and/or performance if there's a problem with the NFS= server? Or the network?

Thanks,
randy


On 01/14/2013 11:36 PM, Harsh J wrote:
Its very rare to observe an NN crash due to a software bu= g in
production. Most of the times its a hardware fault you should worry about.<= br>
On 1.x, or any non-HA-carrying release, the best you can get to
safeguard against a total loss is to have redundant disk volumes
configured, one preferably over a dedicated remote NFS mount. This way
the NN is recoverable after the node goes down, since you can retrieve a current copy from another machine (i.e. via the NFS mount) and set a new node up to replace the older NN and continue along.

A load balancer will not work as the NN is not a simple webserver - it
maintains state which you cannot sync. We wrote HA-HDFS features to
address the very concern you have.

If you want true, painless HA, branch-2 is your best bet at this point.
An upcoming 2.0.3 release should include the QJM based HA features that
is painless to setup and very reliable to use (over other options), and
works with commodity level hardware. FWIW, we've (my team and I) been
supporting several users and customers who're running the 2.x based HA
in production and other types of environments and it has been greatly
stable in our experience. There are also some folks in the community
running 2.x based HDFS for HA/else.


On Tue, Jan 15, 2013 at 6:55 AM, Panshul Whisper <ouchwhisper@gmail.com
<mailto:ouchwhisper@gmail.com>> wrote:

    Hello,

    Is there a standard way to prevent the failure of Namenode cr= ash in
    a Hadoop cluster?
    or what is the standard or best practice for overcoming the S= ingle
    point failure problem of Hadoop.

    I am not ready to take chances on a production server with Ha= doop
    2.0 Alpha release, which claims to have solved the problem. A= re
    there any other things I can do to either prevent the failure= or
    recover from the failure in a very short time.

    Thanking You,

    --
    Regards,
    Ouch Whisper
    010101010101




--
Harsh J




--
Harsh J
--_000_C2496325850AA74C92AAF83AA9662D2630E335B8szxeml531mbxchi_--