hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: Does io.sort.mb count in the records or just the keys?
Date Sun, 09 Nov 2014 21:03:37 GMT
It accounts for both keys and values.

Hortonworks Inc.

On Sun, Nov 9, 2014 at 11:54 AM, Muhuan Huang <mhhuang@cs.ucla.edu> wrote:

> Hello everyone,
> I have a question about the io.sort.mb property. The document says that
> io.sort.mb is the total amount of buffer memory to use while sorting files.
> My question is that does it include both the keys and values of the records
> or just keys (and perhaps some pointers to the values)?
> More specifically in the case of terasort where each record is 100 bytes
> but the key is only 10 bytes, if io.sort.mb is set to 100, does it mean
> that it can support a maximum of 1M records or 10M records?
> Thanks a lot!
> Muhuan

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message