Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D158BDD58 for ; Mon, 19 Nov 2012 15:21:39 +0000 (UTC) Received: (qmail 92054 invoked by uid 500); 19 Nov 2012 15:21:35 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 91935 invoked by uid 500); 19 Nov 2012 15:21:34 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 91928 invoked by uid 99); 19 Nov 2012 15:21:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Nov 2012 15:21:34 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dontariq@gmail.com designates 209.85.216.41 as permitted sender) Received: from [209.85.216.41] (HELO mail-qa0-f41.google.com) (209.85.216.41) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Nov 2012 15:21:30 +0000 Received: by mail-qa0-f41.google.com with SMTP id c26so61083qad.14 for ; Mon, 19 Nov 2012 07:21:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=jN3mJHKs15IAFbcqDHbAvSEf/E7FfIckhbo/GTKcLxc=; b=0a2khJy/Y3Sne3jDw8T+9pmJveXAfC6HoLmfTK39cfvX5fMnOh6byitwSkdns1oapj /ITtdd76YHQlvI1utw5IRCuEFNRze34aujtn0ybO+VYBloW1r2SsYFf1BiAa5w7yIuhP KlHP8WbZWCgifB5QvYA817y/AEP5w5gOxTC54DmwxUL928SIHIyfkzsbZVLGJJj85sXT bgfmHvhPGlTY4fUHhWycyFx9Azp4VujvZwB9GVUSs/V5YG5gqgFwz3GRZiM9o26hmvvE SAXF0xWATUTA+ndOv+XPkMqjbp+qJyuxNSlFFrzI+DcL3jXvyYiJhr7Q4zseenPnO1mY HLOw== Received: by 10.224.182.142 with SMTP id cc14mr11711504qab.23.1353338469537; Mon, 19 Nov 2012 07:21:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.229.183.84 with HTTP; Mon, 19 Nov 2012 07:20:29 -0800 (PST) In-Reply-To: References: <106F2F9A-2A45-4B79-B392-4BBCCB2B04E5@123.org> <5DD097F1-A97A-4C36-B8EF-2CB549EE32DB@123.org> From: Mohammad Tariq Date: Mon, 19 Nov 2012 20:50:29 +0530 Message-ID: Subject: Re: a question on NameNode To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=20cf303b3e7fbcd82904cedaaa95 X-Virus-Checked: Checked by ClamAV on apache.org --20cf303b3e7fbcd82904cedaaa95 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hello Andy, If you have not disabled the speculative execution then your second assumption is correct. Regards, Mohammad Tariq On Mon, Nov 19, 2012 at 8:44 PM, Kartashov, Andy wr= ote: > Thank you Kai.. One more question please. > > > > Does MapReduce run tasks of redundant blocks ? > > > > Say you have only 1 block of data replicated 3 times, one block over each > of three DNodes, block 1 =96 DN1 / block 1(replica #1) =96 DN2 / block1 > (replica #2) =96 DN3 > > > > Will MR attempt: > > > > a. to start 3 Map tasks (one per replicated block) end execute them > all > > b. to start 3 Map tasks (one per replicated block) end drop the > other two as soon as one of the three executed successfully > > c. will start only 1 Map task (for just one block avoiding all > replicated ones) and will attempt to start (another one of the replicated > blocks) when and only when the initially task running (say on DN1)failed > > > > Thanks, > > > > *From:* Kai Voigt [mailto:k@123.org] > *Sent:* Monday, November 19, 2012 10:01 AM > > *To:* user@hadoop.apache.org > *Subject:* Re: a question on NameNode > > > > > > Am 19.11.2012 um 15:43 schrieb "Kartashov, Andy" = : > > > > So, what if DN2 is down, i.e. it is not sending any blocks=92 report. > Then NN (I guess) will figure out that it has 2 blocks (3,4) that has no > home and that (without replication) it has no way of reconstructing the > file A.txt. It must spit the error then. > > > > One major feature of HDFS is its redundancy. Blocks are stored more than > once (three times by default), so chances are good that another DataNode > will have that block and report it during the safe mode phase. So the fil= e > will be accessible. > > > > Kai > > > > -- > > Kai Voigt > > k@123.org > > > > > > > NOTICE: This e-mail message and any attachments are confidential, subjec= t > to copyright and may be privileged. Any unauthorized use, copying or > disclosure is prohibited. If you are not the intended recipient, please > delete and contact the sender immediately. Please consider the environmen= t > before printing this e-mail. AVIS : le pr=E9sent courriel et toute pi=E8c= e > jointe qui l'accompagne sont confidentiels, prot=E9g=E9s par le droit d'a= uteur > et peuvent =EAtre couverts par le secret professionnel. Toute utilisation= , > copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas = le > destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diat= ement > l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'imprimer le p= r=E9sent > courriel > --20cf303b3e7fbcd82904cedaaa95 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hello Andy,

=A0 =A0 If you have not disabled the specula= tive execution then your second assumption is correct.

Regards,
=A0=A0 =A0Mohammad Tariq


On Mon, Nov 19, 2012 at 8:44 PM, Kartash= ov, Andy <Andy.Kartashov@mpac.ca> wrote:

Thank you Kai.. One more = question please.

=A0

Does MapReduce run tasks = of redundant blocks ?

=A0

Say you have only 1 block= of data replicated 3 times, one block over each of three DNodes, block 1 = =96 DN1 / block 1(replica #1) =96 DN2 / block1 (replica #2) =96 DN3

=A0

Will MR attempt:

=A0

a.=A0=A0=A0=A0=A0=A0 to start 3 Map tasks (one p= er replicated block) end execute them all

b.=A0=A0=A0=A0=A0 to start 3 Map tasks (one p= er replicated block) end drop the other two as soon as one of the three exe= cuted successfully

c.=A0=A0=A0=A0=A0=A0 will start only 1 Map task = (for just one block avoiding all replicated ones) and will attempt to start= (another one of the replicated blocks) when and only when the initially task running (say on DN1)failed

=A0

Thanks,

=A0

From: Kai Voigt [mailto:k@123.org]
Sent: Monday, November 19, 2012 10:01 AM


To: user= @hadoop.apache.org
Subject: Re: a question on NameNode

=A0

=A0

Am 19.11.2012 um 15:43 schrieb "Kartashov, Andy= " <Andy= .Kartashov@mpac.ca>:



So, what if DN2 is down, = i.e. it is not sending any blocks=92 report.=A0 Then NN (I guess) will figu= re out that it has 2 blocks (3,4) that has no home and that (without replication) it has no way of reconstructing the file A.txt. It m= ust spit the error then.

=A0

One major feature of HDFS is its redundancy. Blocks = are stored more than once (three times by default), so chances are good tha= t another DataNode will have that block and report it during the safe mode = phase. So the file will be accessible.

=A0

Kai

=A0

--=A0

Kai Voigt

=A0



=A0

NOTICE: This e-mail message and any attachments are confidential, subject t= o copyright and may be privileged. Any unauthorized use, copying or disclos= ure is prohibited. If you are not the intended recipient, please delete and= contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr= =E9sent courriel et toute pi=E8ce jointe qui l'accompagne sont confiden= tiels, prot=E9g=E9s par le droit d'auteur et peuvent =EAtre couverts pa= r le secret professionnel. Toute utilisation, copie ou divulgation non autoris=E9e est interdite. Si vous n'=EAtes pas le = destinataire pr=E9vu de ce courriel, supprimez-le et contactez imm=E9diatem= ent l'exp=E9diteur. Veuillez penser =E0 l'environnement avant d'= ;imprimer le pr=E9sent courriel

--20cf303b3e7fbcd82904cedaaa95--