Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D21B4173D7 for ; Sun, 5 Apr 2015 05:56:42 +0000 (UTC) Received: (qmail 85843 invoked by uid 500); 5 Apr 2015 05:56:31 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 85731 invoked by uid 500); 5 Apr 2015 05:56:31 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 85721 invoked by uid 99); 5 Apr 2015 05:56:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Apr 2015 05:56:31 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of daemeonr@gmail.com designates 209.85.223.174 as permitted sender) Received: from [209.85.223.174] (HELO mail-ie0-f174.google.com) (209.85.223.174) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Apr 2015 05:56:05 +0000 Received: by ierf6 with SMTP id f6so3414533ier.2 for ; Sat, 04 Apr 2015 22:54:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ZbpjPje7ZecpvGUkHcapTegyLX7+4kxuooG5lK1dxAQ=; b=pQaitrBd7UC7LhhUleb1oFnKEl08AyvP/2bg97NYH0ze6q9+3+sqSf1JbcUK6+jn6y LNQR6mp2vPUciGQOuW11BIpZhHCH9/DLbEuuKr+3KjgnIMEFjHRLpKd1Z72GiVeveN6a 5IbMF8SZlSJLldlrbf3NliKm0389Twfhww7KvqAHmL47trSBhuG2YQ/4+OpoPGlOso7V K43rv0+PiGMzXtEl207yc7SRFYe+KX+i6d2cRI2ui1PK+yOx68CUCer0yNSnLS1HTPgu CxmIe8831nosxYcgCtpgvtFt8sQc0Ks/6gV6T/PcawTy++ufbozUydDKBVh7JH35GIpT B9WQ== MIME-Version: 1.0 X-Received: by 10.107.162.74 with SMTP id l71mr13870596ioe.77.1428213273198; Sat, 04 Apr 2015 22:54:33 -0700 (PDT) Received: by 10.50.216.229 with HTTP; Sat, 4 Apr 2015 22:54:33 -0700 (PDT) Received: by 10.50.216.229 with HTTP; Sat, 4 Apr 2015 22:54:33 -0700 (PDT) In-Reply-To: References: Date: Sat, 4 Apr 2015 22:54:33 -0700 Message-ID: Subject: Re: How will Hadoop handle it when a datanode server with total hardware failure? From: daemeon reiydelle To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1140ca38d005300512f3d084 X-Virus-Checked: Checked by ClamAV on apache.org --001a1140ca38d005300512f3d084 Content-Type: text/plain; charset=UTF-8 With rep of 3 you would have to lose 3 entire nodes to lose data. The rep factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine how hdfs spreads io across the spindles for the single copy of the data (one of 3 nodes with copies) that the node owns. Note that things get slightly complicated when the FIRST datum is written to a cluster. (But that was not your question ; {) On Apr 4, 2015 10:39 PM, "Arthur Chan" wrote: > Hi, > > I use the default replication factor 3 here, the cluster has 10 nodes, > each of my datanode has 8 hard disks. If one of the nodes is down because > of hardware failure, i.e. the 8 hard disks will no longer be available > immediately during the down time of this machine, does it mean that I will > have data lost? (8 hard disks > 3 replicated) > > Or what would be the maximum number of servers that are allowed to be down > without data lost here? > > Regards > Arthur > > On Wednesday, December 17, 2014, Harshit Mathur > wrote: > >> Hi Arthur, >> >> In HDFS there will be block level replication, In case of total failure >> of a datanode the lost blocks will get under replicated hence the namenode >> will create copy of these under replicated blocks on some other datanode. >> >> BR, >> Harshit >> >> On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com < >> arthur.hk.chan@gmail.com> wrote: >>> >>> Hi, >>> >>> If each of my datanode servers has 8 hard disks (a 10-node cluster) and >>> I use the default replication factor of 3, how will Hadoop handle it when a >>> datanode with total hardware failure suddenly? >>> >>> Regards >>> Arthur >>> >> >> >> >> -- >> Harshit Mathur >> > --001a1140ca38d005300512f3d084 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

With rep of 3 you would have to lose 3 entire nodes to lose = data. The rep factor is 3 nodes, not 3 spindles.. The number of disks (sort= of) determine how hdfs spreads io across the spindles for the single copy = of the data (one of 3 nodes with copies) that the node owns. Note that thin= gs get slightly complicated when the FIRST datum is written to a cluster. (= But that was not your question ; {)

On Apr 4, 2015 10:39 PM, "Arthur Chan"= <arthur.hk.chan@gmail.com> wrote:
Hi,=C2= =A0

I use the default replication factor 3 here, the c= luster has 10 nodes, each of my datanode has 8 hard disks.=C2=A0 If one of = the nodes is down because of hardware failure, i.e. the 8 hard disks will n= o longer be available immediately during the down time of this machine, doe= s it mean that I will have data lost? (8 hard disks > =C2=A03 replicated= )

Or what would be the maximum number of = servers that are allowed to be down without data lost here?=C2=A0

Regards
Arthur

On Wednesday, December 17, 2014, Harshit Mat= hur <mathursh= arp@gmail.com> wrote:
Hi Arthur,

In HDFS there will be block level re= plication, In case of total failure of a datanode the lost blocks will get = under replicated hence the namenode will create copy of these under replica= ted blocks on some other datanode.

BR,
Hars= hit

On Wed, Dec 17, 2014 at 11:35 AM, Arthur.hk.chan@gmail.com <arthur.hk.chan@gmail.com> wrote:Hi,

If each of=C2=A0 my datanode servers has 8 hard disks (a 10-node cluster) a= nd I use the default replication factor of 3, how will Hadoop handle it whe= n a datanode with total hardware failure suddenly?

Regards
Arthur
=C2=A0


--
Ha= rshit Mathur
--001a1140ca38d005300512f3d084--