hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John George <john...@yahoo-inc.com>
Subject Re: a file can be used as a queue?
Date Mon, 13 Jun 2011 14:52:25 GMT



On 6/13/11 6:23 AM, "Joey Echeverria" <joey@cloudera.com> wrote:

> This feature doesn't currently work. I don't remember the JIRA for it, but
> there's a ticket which will allow a reader to read from an HDFS file before
> it's closed. In that case, you implement a queue by having the producer write
> to the end of the file and the reader read from the beginning of the file.
> 
> I'm not sure if there will be a way to tell that a file is still being
> written, so you may need your own end of stream marker.

One way to know the end of stream would be to call getVisibleLength() on the
input stream. As long as the writer has flushed (or closed) its stream, the
reader should be able to see those bytes. TestWriteRead.java might provide
you some clues  
(hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestWriteRead.java).

> 
> -Joey
> 
> On Jun 13, 2011, at 2:55, ltomuno <ltomuno@163.com> wrote:
> 
>> I heard a HDFS file as a producer - consumer queue, a file can be used as a
>> queue? I am very confused

Regards,
John George


Mime
View raw message