Return-Path: Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: (qmail 51749 invoked from network); 28 Oct 2010 05:58:12 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 28 Oct 2010 05:58:12 -0000 Received: (qmail 94001 invoked by uid 500); 28 Oct 2010 05:58:12 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 93848 invoked by uid 500); 28 Oct 2010 05:58:12 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 93841 invoked by uid 500); 28 Oct 2010 05:58:12 -0000 Delivered-To: apmail-hadoop-core-commits@hadoop.apache.org Received: (qmail 93838 invoked by uid 99); 28 Oct 2010 05:58:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 05:58:12 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.131] (HELO eos.apache.org) (140.211.11.131) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 05:58:10 +0000 Received: from eosnew.apache.org (localhost [127.0.0.1]) by eos.apache.org (Postfix) with ESMTP id C54DF1D6; Thu, 28 Oct 2010 05:57:33 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Apache Wiki To: Apache Wiki Date: Thu, 28 Oct 2010 05:57:33 -0000 Message-ID: <20101028055733.75229.90919@eosnew.apache.org> Subject: =?utf-8?q?=5BHadoop_Wiki=5D_Update_of_=22HDFS-RAID=22_by_RamkumarVadali?= X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch= ange notification. The "HDFS-RAID" page has been changed by RamkumarVadali. http://wiki.apache.org/hadoop/HDFS-RAID?action=3Ddiff&rev1=3D5&rev2=3D6 -------------------------------------------------- bytes. We can recover any 3 missing bytes by the other 10 remaining bytes. = There are two kinds of erasure codes implemented in Raid: XOR code and Re= ed-Solomon code. The difference between them is that XOR only allows creati= ng one parity - bytes but Reed-Solomon code allows creating any given number of parity by= tes. + bytes but Reed-Solomon code allows creating any given number of parity by= tes. As a result, the replication on the source file can be reduce to 1 whe= n using Reed-Solomon + without losing data safety. The downside of having only one replica of a = block is that reads of a block have to go to a single machine, reducing par= allelism. Thus + Reed-Solomon should be used on data that is not supposed to be used frequ= ently. = =3D=3D Using HDFS RAID =3D=3D = @@ -99, +101 @@ =3D=3D=3D Configuration =3D=3D=3D = There is a single configuration file named `raid.xml` that describes the = HDFS - paths for which RAID should be used. A sample of this file can be found in - `src/contrib/raid/conf/raid.xml`. To apply the policies defined in `raid.= xml`, = - a reference has to be added to `hdfs-site.xml`: + paths for which RAID should be used. This provides a list of directory/fi= le patterns + that need to be RAIDed. There are quite a few options that can be specifi= ed for + each pattern. A sample of this file can be found in`src/contrib/raid/conf= /raid.xml`. + To apply the policies defined in `raid.xml`, a reference has to be added = to `hdfs-site.xml`: {{{ raid.config.file