Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of e.v.skaley@gmail.com
 designates 209.85.214.50 as permitted sender)
Message-ID: <50DAE37E.6000709@gmail.com>
Date: Wed, 26 Dec 2012 12:46:06 +0100
From: Eduard Skaley <e.v.skaley@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: user@hadoop.apache.org
CC: Harsh J <harsh@cloudera.com>
Subject: Re: Map Shuffle Bytes
References: <50D99D23.9030902@gmail.com>
 <CAOcnVr1V3A32XoqE9K9ND_mDdPb+0JHsNtZ1RZR0Y58K91xskw@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr1V3A32XoqE9K9ND_mDdPb+0JHsNtZ1RZR0Y58K91xskw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I mean TO the mappers. I'm using the CompositeInputFormat for my 
application to compute map-side joins.
I want to join two datasets A and B one is stored on node 1 and the 
other one on node 2.
For example if the join will be computed on node 2 then the inputsplit 
of the dataset which is stored on node 1 has to be transferred to node 2.
I want to count the bytes which are shuffled (transferred) TO the mapper 
of node 2.
> Hi,
>
> What do you mean by "shuffled bytes [to] the mappers"? If you mean
> "from", it is "Reduce shuffle bytes" you look for; otherwise, you may
> be looking for the per-map counter of "Map output bytes".
>
> Per-partition counters can be constructed on the user side if needed,
> by pre-computing the partition before emit (using the same
> partitioner) and counting up the bytes of your objects for its
> counter.
>
> On Tue, Dec 25, 2012 at 6:03 PM, Eduard Skaley <e.v.skaley@gmail.com> wrote:
>> Hello guys,
>>
>> I need a counter for shuffled bytes to the mappers.
>> Is there existing one or should I define one myself ?
>> How can I implement such a counter?
>>
>> Thank you and happy Christmas time,
>> Eduard
>
>