hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: a question on NameNode
Date Mon, 19 Nov 2012 15:14:44 GMT
Thank you Kai.. One more question please.

Does MapReduce run tasks of redundant blocks ?

Say you have only 1 block of data replicated 3 times, one block over each of three DNodes,
block 1 - DN1 / block 1(replica #1) - DN2 / block1 (replica #2) - DN3

Will MR attempt:

a.       to start 3 Map tasks (one per replicated block) end execute them all

b.      to start 3 Map tasks (one per replicated block) end drop the other two as soon as
one of the three executed successfully

c.       will start only 1 Map task (for just one block avoiding all replicated ones) and
will attempt to start (another one of the replicated blocks) when and only when the initially
task running (say on DN1)failed


From: Kai Voigt [mailto:k@123.org]
Sent: Monday, November 19, 2012 10:01 AM
To: user@hadoop.apache.org
Subject: Re: a question on NameNode

Am 19.11.2012 um 15:43 schrieb "Kartashov, Andy" <Andy.Kartashov@mpac.ca<mailto:Andy.Kartashov@mpac.ca>>:

So, what if DN2 is down, i.e. it is not sending any blocks' report.  Then NN (I guess) will
figure out that it has 2 blocks (3,4) that has no home and that (without replication) it has
no way of reconstructing the file A.txt. It must spit the error then.

One major feature of HDFS is its redundancy. Blocks are stored more than once (three times
by default), so chances are good that another DataNode will have that block and report it
during the safe mode phase. So the file will be accessible.


Kai Voigt

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe
qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite.
Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement
l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

View raw message