hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: map-reduce on a none closed files
Date Sun, 04 Mar 2012 16:49:10 GMT
Technically, yes, you can run MR jobs on non-closed files (It'll run
the reader in the same way as your -cat) , but your would only be able
to read until the last complete block, or until the point sync() was
called on the output stream.

It is better if your file-writer uses the sync() API judiciously to
mark sync points after a considerable amount of records, so that your
MR readers in tasks read until whole records and not just block

For a description on sync() API, read the section 'Coherency Model' in
Tom White's "Hadoop: The Definitive Guide" (O'Reilly), Page 68.

On Sun, Mar 4, 2012 at 8:07 PM, Niv Mizrahi <nivm@taykey.com> wrote:
> hi all,
>  we are looking for a way, to map-reduce on a non-closed files.
>  we currently able to run a
> hadoop fs -cat <non-closed-file>
> non-closed files - files that are currently been written, and have not been
> closed yet.
> is there any way to run map-reduce a on non-closed files ??
> 10x in advance for any answer
> --
> Niv Mizrahi
> Taykey | www.taykey.com

Harsh J

View raw message