Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 14423 invoked from network); 17 Jan 2010 17:27:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 17 Jan 2010 17:27:04 -0000 Received: (qmail 61795 invoked by uid 500); 17 Jan 2010 17:27:02 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 61731 invoked by uid 500); 17 Jan 2010 17:27:01 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 61721 invoked by uid 99); 17 Jan 2010 17:27:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Jan 2010 17:27:01 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of stas.oskin@gmail.com designates 209.85.220.213 as permitted sender) Received: from [209.85.220.213] (HELO mail-fx0-f213.google.com) (209.85.220.213) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Jan 2010 17:26:53 +0000 Received: by fxm5 with SMTP id 5so1608297fxm.29 for ; Sun, 17 Jan 2010 09:26:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type; bh=7Clx/KsGKabvJPr92gh7eSOga7pe0vlZkB/BnrWZfN8=; b=Pl8lfWgLWQtnRSZQVrHXaWZtcQ11lkdCTIN+Cv1hLcVy0oQeEgrP7g+86A+UOC845p bgGRGs9dZndlU48Tb4BbbG4D9jZHpoLxw4qVw/C7KfzK6fNs2m10yi4cM5RcaGaFuBlA OCJ0yaPMnUxqYluMPtQ5H0+c1tjRX1J4XuUi4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=JB2A0WMb64nvFmv87erQx4RRCmWG3bzDfwgEdKbyN9AgSnqTKETkIko/7qyEn1fb2i h/oe759sRE8N7PlWQTAQ9Ekni/SYeJezXZ66oQ3MThYAMzThekfenpKik0QcMUZ2vOg9 NWBaPieam8ZumqQUmbSockC/xUZDrWSXs+Mp4= MIME-Version: 1.0 Received: by 10.223.68.155 with SMTP id v27mr3364325fai.10.1263749193163; Sun, 17 Jan 2010 09:26:33 -0800 (PST) In-Reply-To: References: <77938bc20904091545x623893f6jef73eaa4cac429f0@mail.gmail.com> <77938bc20904101151r23f10826ie1559e6fe9192d7@mail.gmail.com> <49DFBAE5.201@yahoo-inc.com> <77938bc20912281101w11e3cd04r9d5db31784df68c8@mail.gmail.com> <77938bc21001021454k347c76a7o9c59589b8ea386e@mail.gmail.com> <1262488879.9296.1051.camel@andidesk.lan> <1da12e811001030518u680c9a0au61521cbc9bbf6db3@mail.gmail.com> <77938bc21001050559x7e7cf9d0rc8b47e3cd8d1f173@mail.gmail.com> From: Stas Oskin Date: Sun, 17 Jan 2010 19:26:13 +0200 Message-ID: <77938bc21001170926we91ea1p774fd6caed8849b7@mail.gmail.com> Subject: Re: HDFS read/write speeds, and read optimization To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001517475b50be662b047d5f89e6 X-Virus-Checked: Checked by ClamAV on apache.org --001517475b50be662b047d5f89e6 Content-Type: text/plain; charset=ISO-8859-1 Hi. We run with 2-way replication. The wonderful folks at Yahoo! worked through > most of the bugs during 0.19.x IIRC. There was never any bugs with 2-way > replication per-se, but running a cluster with 2 replicas exposed other bugs > at a 100x rate compared to running with 3 replicas (due to the fact that a > silent corruption + loss of a single data node = file loss). > > I'd estimate we lose files at a rate of about 1 per month for 200TB of > actual data. That number would probably go down an order of magnitude or > more if we were running with 3 replicas. > > Hope this helps. > > Thanks for sharing! So, there is a good reason to believe, that version 0.19 and higher have the file storage / silent corruption issues sorted out? Regards. --001517475b50be662b047d5f89e6--