Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of zhengda1936@gmail.com
 designates 209.85.160.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABZdq=Z0hp26PhGOH1Gnii0dRhvEdpGuvgCxF1GW0-yZfRJe2g@mail.gmail.com>
References: 
 <CAFLer82F4YKVzmS__CYhscsCrOen3N0sW10ucQRa3YHyRUwfiQ@mail.gmail.com>
	<CABZdq=Z0hp26PhGOH1Gnii0dRhvEdpGuvgCxF1GW0-yZfRJe2g@mail.gmail.com>
Date: Mon, 31 Oct 2011 15:50:08 -0400
Message-ID: 
 <CAFLer80t4WTFB9D22TcUehsmYVqj7EO9kAjYeUJ0KBUamk1vgA@mail.gmail.com>
Subject: Re: replication in HDFS
From: Zheng Da <zhengda1936@gmail.com>
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hello Ram,

Sorry, I didn't notice your reply.

I don't really have a complete design in my mind. I wonder if the
community is interested in using an alternative scheme to support data
reliability and if the community plans to do it.

You are right, we might need to buffer the source blocks on the local
disk, and using parity blocks might not gain much advantage when we
try to achieve the same reliability as achieved with a small
replication factor. I think in the larger HDFS cluster we need a large
replication factor (>= 3), right? Furthermore, network bandwidth is
scarce resource, so it's more important to save network bandwidth.

Thanks,
Da

On Tue, Oct 25, 2011 at 12:51 AM, Ramkumar Vadali
<ramkumar.vadali@gmail.com> wrote:
> (sorry for the delay in replying)
>
> Hi Zheng
>
> You are right about HDFS RAID. It is used to save space, and is not involved
> in the file write path. The generation of parity blocks and reducing
> replication factor happens after a configurable amount of time.
>
> What is the design you have in mind? When the HDFS file is being written,
> the data is generated block-by-block. But generating parity blocks will
> require multiple source blocks to be ready, so the writer will need to
> buffer the original data, either in memory or on disk. If it is saved on
> disk because of memory pressure, will this be similar to writing the file
> with replication 2?
>
> Ram
>
>
> On Thu, Oct 13, 2011 at 1:16 AM, Zheng Da <zhengda1936@gmail.com> wrote:
>
>> Hello all,
>>
>> Right now HDFS is still using simple replication to increase data
>> reliability. Even though it works, it just wastes the disk space,
>> network and disk bandwidth. For data-intensive applications (that
>> needs to write large result to the HDFS), it just limits the
>> throughput of MapReduce. Also it's very energy-inefficient.
>>
>> Is the community trying to use erasure code to increase data
>> reliability? I know someone is working on HDFS-RAID, but it can only
>> solve the problem in disk space. In many case, network and disk
>> bandwidth are more important, which are the factors limiting the
>> throughput of MapReduce. Has anyone tried to use erasure code to
>> reduce the size of data when data is written to HDFS? I know reducing
>> replications might hurt the read performance, but I think it's still
>> important to reduce the data size writing to HDFS.
>>
>> Thanks,
>> Da
>>
>