Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0395976D6 for ; Tue, 20 Sep 2011 13:50:19 +0000 (UTC) Received: (qmail 62944 invoked by uid 500); 20 Sep 2011 13:50:15 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 62878 invoked by uid 500); 20 Sep 2011 13:50:15 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 62842 invoked by uid 99); 20 Sep 2011 13:50:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Sep 2011 13:50:15 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ajit.ratnaparkhi@gmail.com designates 209.85.210.176 as permitted sender) Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Sep 2011 13:50:11 +0000 Received: by iabz7 with SMTP id z7so871445iab.35 for ; Tue, 20 Sep 2011 06:49:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=9H0Bj67yiNTEZwpsRho52HumPnd+bDoTaALZV6ZYjhg=; b=iped7fbenxAG8Y1e1e9lFNsTAVhd35dQDta6ueZQibQjwoy65veD5AmV8j81SMG21w Vs9c7HP3J8U6iMQbZG/ajpJZU++FCteB8nFVF1cPEgM0CO+lUsoPFpDvJLIBdwE7j+yI aYdpiMN+oK/6MwHCMXNseWIRNAHHx7q/rQlEA= Received: by 10.231.82.12 with SMTP id z12mr1405925ibk.36.1316526589246; Tue, 20 Sep 2011 06:49:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.105.133 with HTTP; Tue, 20 Sep 2011 06:49:29 -0700 (PDT) In-Reply-To: References: <1316106528.69913.YahooMailNeo@web65514.mail.ac4.yahoo.com> <1316276206.37108.YahooMailNeo@web65506.mail.ac4.yahoo.com> From: Ajit Ratnaparkhi Date: Tue, 20 Sep 2011 19:19:29 +0530 Message-ID: Subject: Re: Need help regarding HDFS-RAID To: hdfs-user@hadoop.apache.org Cc: Andrew Purtell Content-Type: multipart/alternative; boundary=000e0cdf15ecb07d8304ad5fbb70 --000e0cdf15ecb07d8304ad5fbb70 Content-Type: text/plain; charset=ISO-8859-1 Thanks Dhruba! Can I try using it? Is it open for use? -Ajit. On Tue, Sep 20, 2011 at 2:48 PM, Dhruba Borthakur wrote: > Hi andy, > > we do run a version of HDFS RAID that is backported from Apache trunk to a > 0.20 based release. Our code is in > https://github.com/facebook/hadoop-20-warehouse/tree/master/src/contrib/raid > But I do not have an elegant way to contribute this code to Apache > 0.20.2xx.x. > > thanks, > dhruba > > > On Sat, Sep 17, 2011 at 9:16 AM, Andrew Purtell wrote: > >> Hi Dhruba, >> >> Would you consider a contribution of this to branch-0.20-security aka >> 0.20.2xx.x? >> >> If I am mistaken and you do not have a 0.22-ish HDFS RAID backported to an >> 0.20-ish platform, please disregard. >> >> Best regards, >> >> - Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >> >> ------------------------------ >> *From:* Dhruba Borthakur >> *To:* hdfs-user@hadoop.apache.org; Andrew Purtell >> *Sent:* Thursday, September 15, 2011 10:14 AM >> >> *Subject:* Re: Need help regarding HDFS-RAID >> >> That's right Andy. 0.22+. We are running a HDFS-RAID code base that is >> pretty close to what is available in Apache hdfs trunk. >> >> -dhruba >> >> On Thu, Sep 15, 2011 at 10:08 AM, Andrew Purtell wrote: >> >> But that is the HDFS RAID effectively in 0.22+, not 0.21, right Dhruba? >> >> Best regards, >> >> - Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >> >> ------------------------------ >> *From:* Dhruba Borthakur >> *To:* hdfs-user@hadoop.apache.org >> *Sent:* Thursday, September 15, 2011 10:06 AM >> *Subject:* Re: Need help regarding HDFS-RAID >> >> We use HDFS RAID in a big way. Data older than 12 days are RAIDED using >> XOR encoding (effective replication of 2.5). Data older than a few months >> are raided using ReedSolomon (effective observed replication factor of 1.5). >> This is running on our 60 PB size cluster for about an year now. >> >> thanks >> dhruba >> >> On Thu, Sep 15, 2011 at 5:31 AM, Ajit Ratnaparkhi < >> ajit.ratnaparkhi@gmail.com> wrote: >> >> Hi, >> >> We were planning to use it for past data archival(instead of moving it to >> archival store). >> Archiving it in HDFS gives advantage of making it easily available for >> processing whenever required. >> >> Is there any archival solution in hadoop ecosystem? >> >> thanks, >> Ajit. >> >> >> On Thu, Sep 15, 2011 at 5:05 PM, Harsh J wrote: >> >> Hey Ajit, >> >> HDFS-RAID was never part of the 0.20 release. It made its debut in the >> 0.21 release [1]. I know that Facebook uses it (and also did develop >> it), but unsure of users beyond Facebook. >> >> While 0.21 overall is not entirely deemed as production-usable yet >> (and is in fact, possibly abandoned for efforts on 0.22+), you can >> give that release a whirl on a test cluster and see for yourself if >> your need beats the stability. >> >> Just curious though - why are you looking to use this specifically? >> >> [1] - >> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21/mapreduce/src/contrib/raid/ >> >> On Thu, Sep 15, 2011 at 4:37 PM, Ajit Ratnaparkhi >> wrote: >> > Hi, >> > We want to use HDFS-RAID in our production cluster. >> > (http://wiki.apache.org/hadoop/HDFS-RAID) >> > I am not able to find source/binaries/configs for this in official >> hadoop >> > distribution from apache hadoop. (checked in 0.20.1 and 0.20.2). >> > Can somebody please tell me where can I find that? and installation >> > procedure? >> > Also, is HDFS-RAID implementation stable enough to use in production? >> > thanks, >> > Ajit. >> > >> >> >> >> -- >> Harsh J >> >> >> >> >> >> -- >> Connect to me at http://www.facebook.com/dhruba >> >> >> >> >> >> -- >> Connect to me at http://www.facebook.com/dhruba >> >> >> > > > -- > Connect to me at http://www.facebook.com/dhruba > --000e0cdf15ecb07d8304ad5fbb70 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Dhruba!

Can I try using it? Is it open for use?

-Ajit.

On Tue, Se= p 20, 2011 at 2:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
Hi andy,

we do run a ver= sion of HDFS RAID that is backported from Apache trunk to a 0.20 based rele= ase. Our code is in=A0https://github.com/fa= cebook/hadoop-20-warehouse/tree/master/src/contrib/raid
But I do not have an elegant way to contribute this code to Apache=A0<= span style=3D"font-family:'Courier New', courier, monaco, monospace= , sans-serif;font-size:13px;background-color:rgb(255, 255, 255)">0.20.2xx.x= .=A0

thanks,
dhruba


On Sat, Sep 17, 2011 at 9:16 AM, Andr= ew Purtell <apurtell@apache.org> wrote:
Hi Dhruba,

Wo= uld you consider a contribution of this to branch-0.20-security aka=A00.20.2xx.x?

If I am mistaken and you do not have= a 0.22-ish HDFS RAID backported to an 0.20-ish platform, please disregard.=

Best regards,

=A0 =A0 - Andy
Problems worthy of attack prove their worth by hitting back. - P= iet Hein (via Tom White)
From: Dhruba = Borthakur <dhruba@= gmail.com>
To: hdfs-us= er@hadoop.apache.org; Andrew Purtell <apurtell@apache.org>
Sent: Thursday, September 15= , 2011 10:14 AM

Subject: Re: Need help regarding HDFS-RAID

That's right Andy. 0.22+. We= are running a HDFS-RAID code base that is pretty close to what is availabl= e in Apache hdfs trunk.

-dhruba

On Thu, Sep 15, 2011 at 10:08 AM, Andrew Purtell <apurt= ell@apache.org> wrote:
But that is the HDFS RAID effectively in 0.22+, not 0.21, right = Dhruba?
=A0
Best regards,

= =A0=A0=A0 - Andy

Problems worthy of attack prove their wor= th by hitting back. - Piet Hein (via Tom White)
From: Dhruba Borthakur <dhruba@gmail.com>
To: hdfs-user@hadoop.apach= e.org
Sent: Thursday, September 15= , 2011 10:06 AM
Subject: = Re: Need help regarding HDFS-RAID

We use HDFS RAID in a big way. Data older than 12 days are RAIDED = using XOR encoding (effective replication of 2.5). Data older than a few mo= nths are raided using ReedSolomon (effective observed replication factor of= 1.5). This is running on our 60 PB size cluster for about an year now.
thanks
dhruba

On Thu, Sep 15, 201= 1 at 5:31 AM, Ajit Ratnaparkhi <ajit.ratnaparkhi= @gmail.com> wrote:
Hi,

We were planning to use it for past data ar= chival(instead of moving it to archival store).
Archiving it in HDFS gives advantage of making it easily available for= processing whenever required.

Is there any archival solution in hadoop ecosystem?

thanks,
Ajit.


<= div>On Thu, Sep 15, 2011 at 5:05 PM, Harsh J <harsh@cl= oudera.com> wrote:
Hey Ajit,

HDFS-RAID was never part of the 0.20 release. It made its debut in the
0.21 release [1]. I know that Facebook uses it (and also did develop
it), but unsure of users beyond Facebook.

While 0.21 overall is not entirely deemed as production-usable yet
(and is in fact, possibly abandoned for efforts on 0.22+), you can
give that release a whirl on a test cluster and see for yourself if
your need beats the stability.

Just curious though - why are you looking to use this specifically?

[1] - ht= tp://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21/mapreduce/= src/contrib/raid/

On Thu, Sep 15, 2011 at 4:37 PM, Ajit Ratnaparkhi
<ajit.ratnaparkhi@gmail.com> wrote:
> Hi,
> We want to use HDFS-RAID in our production cluster.
> (http://wiki.apache.org/hadoop/HDFS-RAID)
> I am not able to find source/binaries/configs for this in official had= oop
> distribution from apache hadoop. (checked in 0.20.1 and 0.20.2).
> Can somebody please tell me where can I find that? and installation > procedure?
> Also, is HDFS-RAID implementation stable enough to use in production?<= br> > thanks,
> Ajit.
>



--
Harsh J




--
= Connect to me at http://www.facebook.com/dhruba





--
Connect to me at <= a rel=3D"nofollow" href=3D"http://www.facebook.com/dhruba" target=3D"_blank= ">http://www.facebook.com/dhruba





--
Connect to me at <= a href=3D"http://www.facebook.com/dhruba" target=3D"_blank">http://www.face= book.com/dhruba

--000e0cdf15ecb07d8304ad5fbb70--