Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of brock@cloudera.com designates
 209.85.216.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <DD8E22D7-9B3E-4FC2-9451-2C101A5A699B@cern.ch>
References: <DD8E22D7-9B3E-4FC2-9451-2C101A5A699B@cern.ch>
Date: Sat, 17 Mar 2012 09:50:08 -0500
Message-ID: 
 <CAFukC=4SgECjOS6D7fNXYOnkNohe1yey1sCeSR2w9q6Pnyd79w@mail.gmail.com>
Subject: Re: random seeks during write in HDFS
From: Brock Noland <brock@cloudera.com>
To: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

This question is for hdfs-user not mapreduce-user, as such I have removed them.

Yes HDFS does not allow ramdom writes. I suggest your read this doc:
http://hadoop.apache.org/common/docs/current/hdfs_design.html

Specifically the "Assumptions and Goals" section.

Here are two ways to get around this design assumption:

1) Write updated copies of the record with a new time stamp and then
dedup based on a unique key and timestamp.
2) Use HBase

Cheers,
Brock

On Sat, Mar 17, 2012 at 9:09 AM, Hassen Riahi <hassen.riahi@cern.ch> wrote:
> Hi,
>
> We are trying to execute a mapper making a random access during writing
> files. It seems that HDFS supports only random seek during read and not
> during write (neither the file modification). Is it right? we are using
> hadoop-0.20. If it is the case, is there any plan to support it in the
> future?
>
> The limitation described above makes the mapper failing to write files. Is
> there any suggestions to bypass this limitation? such as write files in a
> temp area and copying them then to HDFS?
>
> Thanks for the help,
> Hassen
>


-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/