hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <danielru...@gmail.com>
Subject Re: Chaining MapReduce
Date Sat, 22 Aug 2015 07:50:50 GMT
Hi,
Data is divided to mappers depending on your inputformat.
Usually the number of mappers = number of blocks.

Daniel

> On 22 באוג׳ 2015, at 09:02, ☼ R Nair (रविशंकर नायर) <ravishankar.nair@gmail.com>
wrote:
> 
> Hi ,
> 
> The mappers depend on source data only. But data definitely is going through all mappers,
so I should get number of map jpbs as my output right? Instead I am getting only one.
> 
> Thanks and regards,
> Ravion
> 
>> On Fri, Aug 21, 2015 at 1:35 PM, ☼ R Nair (रविशंकर नायर)
<ravishankar.nair@gmail.com> wrote:
>> All,
>> 
>> I have three mappers, followed by a reducer. I executed the map reduce successfully.
The reported output shows that number of mappers executed is 1 and number of reducers is also
1. Though number of reducers are correct, won't we be getting number of mappers as 3 , since
I have three mapper classes connected by ChainMapper?
>> 
>> O/P given below (snippet) :-
>> 
>> Job Counters 
>>         Launched map tasks=1
>>         Launched reduce tasks=1
>>         Data-local map tasks=1
>>         Total time spent by all maps in occupied slots (ms)=8853
>>         Total time spent by all reduces in occupied slots (ms)=9900
>>         Total time spent by all map tasks (ms)=8853
>>         Total time spent by all reduce tasks (ms)=9900
>>         Total vcore-seconds taken by all map tasks=8853
>>         Total vcore-seconds taken by all reduce tasks=9900
>>         Total megabyte-seconds taken by all map tasks=9065472
>>         Total megabyte-seconds taken by all reduce tasks=10137600
>> 
>> 
>> What I guess is, since the output is passing through Context, the internal connected
mappers are not caught by job counter, am I correct ?
>> 
>> Best, Ravion
> 

Mime
View raw message