Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.223.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <521BD7B9.90801@plutoz.com>
References: <521BABF7.7040702@plutoz.com>
 <CAOcnVr1avgwj5oJTTFdo7Yd=_ADUXtoD-PWyUjJ1fJaYd0-kfQ@mail.gmail.com>
 <521BD7B9.90801@plutoz.com>
From: Harsh J <harsh@cloudera.com>
Date: Tue, 27 Aug 2013 04:21:38 +0530
Message-ID: 
 <CAOcnVr1rZ_4YWXzWgHdAvJwme2gjLKJMSrtw8_XRQt5Rg9LXkw@mail.gmail.com>
Subject: Re: 1.2.1 ouput, what is bufvoid
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

The length printed there is the # of elements in an internal array,
not a data size length. If you wish to raise that you can raise up
io.sort.spill.percent (default 0.80, or 80%).

On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <kaveh@plutoz.com> wrote:
> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
> only 1mb (1048576)
>
>
> On 08/26/2013 03:14 PM, Harsh J wrote:
>>
>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>
>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <kaveh@plutoz.com> wrote:
>>>
>>> I am getting these in job output in the 1.2.1
>>>
>>> Spilling map output: record full = true
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>> 1048561; kvend = 838844; length = 1048576
>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>> spill 10
>>>
>>>
>>> my question is what is bufvoid? and how can I increase the amount of the
>>> length?
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>>
>>
>>
>>
>
> --
> Kaveh Minooie


-- 
Harsh J