hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajit Ratnaparkhi <ajit.ratnapar...@gmail.com>
Subject Re: Need help regarding HDFS-RAID
Date Tue, 20 Sep 2011 13:49:29 GMT
Thanks Dhruba!

Can I try using it? Is it open for use?

-Ajit.

On Tue, Sep 20, 2011 at 2:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> Hi andy,
>
> we do run a version of HDFS RAID that is backported from Apache trunk to a
> 0.20 based release. Our code is in
> https://github.com/facebook/hadoop-20-warehouse/tree/master/src/contrib/raid
> But I do not have an elegant way to contribute this code to Apache
> 0.20.2xx.x.
>
> thanks,
> dhruba
>
>
> On Sat, Sep 17, 2011 at 9:16 AM, Andrew Purtell <apurtell@apache.org>wrote:
>
>> Hi Dhruba,
>>
>> Would you consider a contribution of this to branch-0.20-security aka
>> 0.20.2xx.x?
>>
>> If I am mistaken and you do not have a 0.22-ish HDFS RAID backported to an
>> 0.20-ish platform, please disregard.
>>
>> Best regards,
>>
>>     - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>>  ------------------------------
>> *From:* Dhruba Borthakur <dhruba@gmail.com>
>> *To:* hdfs-user@hadoop.apache.org; Andrew Purtell <apurtell@apache.org>
>> *Sent:* Thursday, September 15, 2011 10:14 AM
>>
>> *Subject:* Re: Need help regarding HDFS-RAID
>>
>> That's right Andy. 0.22+. We are running a HDFS-RAID code base that is
>> pretty close to what is available in Apache hdfs trunk.
>>
>> -dhruba
>>
>> On Thu, Sep 15, 2011 at 10:08 AM, Andrew Purtell <apurtell@apache.org>wrote:
>>
>> But that is the HDFS RAID effectively in 0.22+, not 0.21, right Dhruba?
>>
>> Best regards,
>>
>>     - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>> ------------------------------
>> *From:* Dhruba Borthakur <dhruba@gmail.com>
>> *To:* hdfs-user@hadoop.apache.org
>> *Sent:* Thursday, September 15, 2011 10:06 AM
>> *Subject:* Re: Need help regarding HDFS-RAID
>>
>> We use HDFS RAID in a big way. Data older than 12 days are RAIDED using
>> XOR encoding (effective replication of 2.5). Data older than a few months
>> are raided using ReedSolomon (effective observed replication factor of 1.5).
>> This is running on our 60 PB size cluster for about an year now.
>>
>> thanks
>> dhruba
>>
>> On Thu, Sep 15, 2011 at 5:31 AM, Ajit Ratnaparkhi <
>> ajit.ratnaparkhi@gmail.com> wrote:
>>
>> Hi,
>>
>> We were planning to use it for past data archival(instead of moving it to
>> archival store).
>> Archiving it in HDFS gives advantage of making it easily available for
>> processing whenever required.
>>
>> Is there any archival solution in hadoop ecosystem?
>>
>> thanks,
>> Ajit.
>>
>>
>> On Thu, Sep 15, 2011 at 5:05 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hey Ajit,
>>
>> HDFS-RAID was never part of the 0.20 release. It made its debut in the
>> 0.21 release [1]. I know that Facebook uses it (and also did develop
>> it), but unsure of users beyond Facebook.
>>
>> While 0.21 overall is not entirely deemed as production-usable yet
>> (and is in fact, possibly abandoned for efforts on 0.22+), you can
>> give that release a whirl on a test cluster and see for yourself if
>> your need beats the stability.
>>
>> Just curious though - why are you looking to use this specifically?
>>
>> [1] -
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21/mapreduce/src/contrib/raid/
>>
>> On Thu, Sep 15, 2011 at 4:37 PM, Ajit Ratnaparkhi
>> <ajit.ratnaparkhi@gmail.com> wrote:
>> > Hi,
>> > We want to use HDFS-RAID in our production cluster.
>> > (http://wiki.apache.org/hadoop/HDFS-RAID)
>> > I am not able to find source/binaries/configs for this in official
>> hadoop
>> > distribution from apache hadoop. (checked in 0.20.1 and 0.20.2).
>> > Can somebody please tell me where can I find that? and installation
>> > procedure?
>> > Also, is HDFS-RAID implementation stable enough to use in production?
>> > thanks,
>> > Ajit.
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>>
>>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

Mime
View raw message