hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Xu <ted.xu...@gmail.com>
Subject Re: Problem while writing a file from Reducer
Date Sat, 05 Dec 2009 01:51:56 GMT
Hello Parth,

Once I met a similar problem while using user-defined recordwriter in
reducer. That time I forgot to close the recordwriter in the close phase of
reducer.

2009/12/5 Parth J. Brahmbhatt <parth@ccs.neu.edu>

> Hello All,
>
> I am creating 2 files in the constructor of my reducer and I store the file
> Handles as member variables. I write some data in these files for each call
> to reduce method. For some reason the files are created but only the larger
> file has data(The larger file is 400mb).The smaller file is created but no
> data is written to it (The size should be around 4MB). I checked the hadoop
> configuration and the buffer-size parameter is 4096.
>
> I padded some 100MB data to the smaller file and than this file gets around
> 64MB data. I am clueless what is happening and if there is a solution to
> this. Please share if you have faced similar problem or if you know the
> solution.
>
> Thanks,
> Parth Brahmbhatt
>

Best Regards,

Ted Xu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message