hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bing Jiang <jiangbinglo...@gmail.com>
Subject Re: produce a large sequencefile (1TB)
Date Tue, 20 Aug 2013 00:14:08 GMT
I think you are worrying about the volumn of mapreduce local file, but
would  you give us more details about your apps.
 On Aug 20, 2013 6:09 AM, "Jerry Lam" <chilinglam@gmail.com> wrote:

> Hi Hadoop users and developers,
> I have a use case that I need produce a large sequence file of 1 TB in
> size when each datanode has  200GB of storage but I have 30 datanodes.
> The problem is that no single reducer can hold 1TB of data during the
> reduce phase to generate a single sequence file even I use aggressive
> compression. Any datanode will run out of space since this is a single
> reducer job.
> Any comment and help is appreciated.
> Jerry

View raw message