Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 76760DE3E for ; Mon, 13 Aug 2012 20:23:56 +0000 (UTC) Received: (qmail 55095 invoked by uid 500); 13 Aug 2012 20:23:56 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 55031 invoked by uid 500); 13 Aug 2012 20:23:56 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 55022 invoked by uid 99); 13 Aug 2012 20:23:56 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Aug 2012 20:23:56 +0000 Received: from localhost (HELO mail-qa0-f41.google.com) (127.0.0.1) (smtp-auth username billie, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Aug 2012 20:23:55 +0000 Received: by qafk30 with SMTP id k30so3125713qaf.0 for ; Mon, 13 Aug 2012 13:23:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.224.28.14 with SMTP id k14mr28085109qac.72.1344889434729; Mon, 13 Aug 2012 13:23:54 -0700 (PDT) Received: by 10.49.98.163 with HTTP; Mon, 13 Aug 2012 13:23:54 -0700 (PDT) In-Reply-To: References: Date: Mon, 13 Aug 2012 16:23:54 -0400 Message-ID: Subject: Re: Hardware failure and data protection From: Billie Rinaldi To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=14dae9cfc39204ee7904c72b790e --14dae9cfc39204ee7904c72b790e Content-Type: text/plain; charset=ISO-8859-1 On Mon, Aug 13, 2012 at 12:31 PM, Aji Janis wrote: > I am very new to Hadoop and Accumulo. I need some information on how data > is backed up or guaranteed against system failures (if it is).I am > considering setting up a Hadoop cluster consisting of 5 nodes where each > node has 3 internal hard drives. I understand HDFS has a configurable > redundancy feature but what happens if an entire drive crashes (physically) > for whatever reason? How does Hadoop recover, if it can, from this > situation? More specifically, I am assuming Accumulo uses HDFS redundancy > to make back ups of the data. > > One, is this assumption true? > Yes, Accumulo uses HDFS replication to preserve data in the presence of failures. HDFS stores N exact copies of each data block, with each copy being stored on a different server. If a drive crashes, HDFS notices that blocks are under-replicated, and copies those blocks to an available drive. Thus the data can survive N-1 simultaneous failures. > Two, if I had a copy of the hard drive and I duplicate that to a new drive > and pop it in where the old/crashed drive used to be would this work? > Since that drive's data would have been replicated to other drives, you should not need a copy of it. You should just be able to put in a fresh hard drive, and HDFS will start using it. Billie > > I apologize if this is a really stupid question. But I highly appreciate > any help, pointers and suggestions! Thanks in advance. > --14dae9cfc39204ee7904c72b790e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, Aug 13, 2012 at 12:31 PM, Aji Janis <aji1705@gmail.com> wrote:
I am very new to Hadoop and Accumulo. I need some information on how data i= s backed up or guaranteed against system failures (if it is).I am considering setting=A0up a Hadoop cluster consisting of 5 nodes= where each node has 3 internal hard drives. I understand HDFS has a config= urable redundancy feature but what happens if an entire drive crashes (phys= ically) for whatever reason? How does Hadoop recover, if it can, from this = situation? More specifically, I am assuming Accumulo uses HDFS redundancy t= o make back ups of the data.=A0

One, is thi= s assumption true?

Yes, Accumulo uses HDFS replication to preserve data = in the presence of failures.=A0 HDFS stores N exact copies of each data blo= ck, with each copy being stored on a different server.=A0 If a drive crashe= s, HDFS notices that blocks are under-replicated, and copies those blocks t= o an available drive.=A0 Thus the data can survive N-1 simultaneous failure= s.
=A0
Two, if I had a copy of the hard drive and I dupl= icate that to a new drive and pop it in where the old/crashed drive used to= be would this work?

Since that drive's data would have been replicate= d to other drives, you should not need a copy of it.=A0 You should just be = able to put in a fresh hard drive, and HDFS will start using it.

Bil= lie
=A0

I apologize if this is a really st= upid question. But I highly appreciate any help, pointers and suggestions! = Thanks in advance.

--14dae9cfc39204ee7904c72b790e--