Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=from:to:references:subject:date:message-id:mime-version:
	content-type:content-transfer-encoding:x-mailer:thread-index:in-reply-to:x-mimeole;
	b=uOwatfbVLH8f9Agaho9dNNdtIEvEAvV476PZsAj1dFLywVgosJ45kRqy62wkjFdz
From: "Devaraj Das" <ddas@yahoo-inc.com>
To: <hadoop-user@lucene.apache.org>
References: <fb1e04d60706260812o326d6946wa42995651c8cbac3@mail.gmail.com>
Subject: RE: is that possible to make MapFile "mutable" ?
Date: Wed, 27 Jun 2007 00:00:15 +0530
Message-ID: <000301c7b820$0ec675a0$2201a8c0@ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Thread-Index: Ace4DZZb3sEWAWlDT6O+8HdHTi1d6QAEg3Vg
In-Reply-To: <fb1e04d60706260812o326d6946wa42995651c8cbac3@mail.gmail.com>

No, you cannot append to a file on the dfs and your app should be able to
treat multiple files as one single logical file (as you point out). But in
your case, it seems like you could design your app to have some buffering,
for example, you could have a buffer for the n different files, and could
flush the buffer to different files on the dfs only when you have reached a
certain limit on the amount of data in the buffer. 
I am not sure whether fault handling is of concern to you but there is the
danger of losing the buffered messages if your app goes down. One way to
handle this - assuming you have the ability to reprocess messages, you could
checkpoint the state of the message processor in the dfs - the state could
include the last message ID you flushed, and the next time your app starts
up, it reads the checkpoint file from the dfs, gets the ID, and process
messages starting from (ID + 1). 

-----Original Message-----
From: Open Study [mailto:open.study@gmail.com] 
Sent: Tuesday, June 26, 2007 8:42 PM
To: hadoop-user@lucene.apache.org
Subject: is that possible to make MapFile "mutable" ?

Hi all,

MapFile doesn't support append mode of creation, so every time the existing
mapfile would be overwritten if a new one with same name is created.

Is there anyway I can append to an MapFile or alike without erasing the old
content? or it doesn't makes sense at all?

In my scenario I need to split mass (count by tens of millions) messages
according to certain rules and put them into different mapfiles, which are
supposed to get updated when new messages come in. Since I didn't find a way
to make mapfile appendable, I have to create new mapfiles, so one mapfile
can contain as little as one message in worst case and I will have to later
merge them with their proper siblings.

Regards