Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B2C48181AD for ; Tue, 8 Mar 2016 23:45:05 +0000 (UTC) Received: (qmail 16141 invoked by uid 500); 8 Mar 2016 23:45:00 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 16023 invoked by uid 500); 8 Mar 2016 23:45:00 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 16009 invoked by uid 99); 8 Mar 2016 23:44:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2016 23:44:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 918071A04D4 for ; Tue, 8 Mar 2016 23:44:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id JYTFVzYw3b8a for ; Tue, 8 Mar 2016 23:44:56 +0000 (UTC) Received: from mail-yk0-f169.google.com (mail-yk0-f169.google.com [209.85.160.169]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 58D865F56B for ; Tue, 8 Mar 2016 23:44:55 +0000 (UTC) Received: by mail-yk0-f169.google.com with SMTP id r203so8246667ykd.3 for ; Tue, 08 Mar 2016 15:44:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to; bh=9YsMKRNFnAIaXXIZ1o16Ik3LSro6oIUFSkvJ/6anCrs=; b=RGvV/3frN6ZgC652JbpYRC6ky9w1u/ny1bPCb7X3GcRipHznu+2Dlek+1otmoLSbce XOqdMuDgAP9+8hpyvWXNRDV4msX/lykRLCXDB8DHMm1DZHa6rkWg3pzi5ul/duzMHk5a 1mj5UAzw0Zkgv0l0+njMWbeUCG30JXU3ecGGf1CLVZ6PlutfVrxXyTrVDmuc6qcwQ8yZ YoDIRxKP2pohEKRapoqryuAK96gmiGkjUiy+7vCFGNSca045UBppqgZN8/Il0NvCNsHw tVx1nOc4FqjnYd8jqcnAoFj8KWr2sCyNNbabLGMcpQZEISc1HVv2dhpnaLJwylltp7br E2QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=9YsMKRNFnAIaXXIZ1o16Ik3LSro6oIUFSkvJ/6anCrs=; b=E2X8JMIgI5pz5rR1fSjvjVfLVN9Q0jWu/Z31SMxXTOOzR6dzWfWf+JcC+2Whfo1srD ozkOMSnys7vUKtz5eLBYeVKjfMGfVZTo0hq7o+GNoeaJEsxKMl6rgN5yfZ4/T5EC61RP 0Vf0wt56vJLnmHtEQ+WDsZYB7cs5juynxByDM0Nhsmhfnfa9v7S+0pyWP2J1ilSvW9H3 +X5+Jxm26PBUj8xWafMlPVDWxrd9R9YlofiTYPgcafL+tiQsYEeyKXKsdLtvBozSi1vw Glwgvhx3o1M69WMAEHBi4NKxmnQdajMGmbUB8Byy6nLOEn9zaXu+9KTYsPph7uLOfSZC F52Q== X-Gm-Message-State: AD7BkJLA4ceeNJAvcX1dWM/x8MdAL20POc9fCGCH5u3A8C4ZkM0K9B7T4B9IcKk5C0CX8JJmjw4jvHVrdxfvGQ== MIME-Version: 1.0 X-Received: by 10.37.33.8 with SMTP id h8mr16676584ybh.175.1457480694201; Tue, 08 Mar 2016 15:44:54 -0800 (PST) Received: by 10.37.215.17 with HTTP; Tue, 8 Mar 2016 15:44:53 -0800 (PST) Date: Tue, 8 Mar 2016 18:44:53 -0500 Message-ID: Subject: Intermediate Data Spill in Mapreduce (Buffer Memory) From: Sultan Alamro To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1143e6900b8a6c052d922b5c --001a1143e6900b8a6c052d922b5c Content-Type: text/plain; charset=UTF-8 Hi there, I run a word count job in Hadoop 2.6.0 and I see that there are several spills for the map output. I have the following configuration: mapreduce.task.io.sort.mb = 100 mapreduce.map.sort.spill.percent = 0.80 After running the job, the Map output bytes = 222660096. By looking at the container log below, it seems that the buffer size is 31055173 bytes not 100MB. Look at the parameters values before spilling the first spill into the disk bufstart = 0; bufend = 31055173; bufvoid = 104857600 And by dividing the Map output bytes by bufend (222660096/31055173 = 7.17 (= 8 spills)). Why is this happening? The buffer size should be 80MB (soft limit), so I think I should have 3 spills only. Also, can anyone give a better explanation for bufstart, bufend and bufvoid? Thanks, Sultan 2016-03-04 20:38:16,678 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hmaster:9000/input/Input-128M:0+134217728 2016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2016-03-04 20:38:16,854 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2016-03-04 20:38:19,931 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:38:19,931 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 31055173; bufvoid = 104857600 2016-03-04 20:38:19,931 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 13006672(52026688); length = 13207725/6553600 2016-03-04 20:38:19,931 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 41540922 kvi 10385224(41540896) 2016-03-04 20:38:28,178 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 0 2016-03-04 20:38:28,179 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 41540922 kv 10385224(41540896) kvi 7763800(31055200) 2016-03-04 20:38:30,020 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:38:30,020 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 41540922; bufend = 72594952; bufvoid = 104857600 2016-03-04 20:38:30,020 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 10385224(41540896); kvend = 23391620(93566480); length = 13208005/6553600 2016-03-04 20:38:30,020 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 83080708 kvi 20770172(83080688) 2016-03-04 20:38:37,042 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 1 2016-03-04 20:38:37,042 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 83080708 kv 20770172(83080688) kvi 18148744(72594976) 2016-03-04 20:38:39,045 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:38:39,045 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 83080708; bufend = 9277592; bufvoid = 104857598 2016-03-04 20:38:39,045 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 20770172(83080688); kvend = 7562280(30249120); length = 13207893/6553600 2016-03-04 20:38:39,045 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 19763348 kvi 4940832(19763328) 2016-03-04 20:38:46,376 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 2 2016-03-04 20:38:46,376 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 19763348 kv 4940832(19763328) kvi 2319404(9277616) 2016-03-04 20:38:48,320 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:38:48,320 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 19763348; bufend = 50820055; bufvoid = 104857600 2016-03-04 20:38:48,320 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 4940832(19763328); kvend = 17947892(71791568); length = 13207341/6553600 2016-03-04 20:38:48,320 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 61305803 kvi 15326444(61305776) 2016-03-04 20:38:55,301 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 3 2016-03-04 20:38:55,301 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 61305803 kv 15326444(61305776) kvi 12705020(50820080) 2016-03-04 20:38:57,209 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:38:57,209 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 61305803; bufend = 92359884; bufvoid = 104857600 2016-03-04 20:38:57,209 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 15326444(61305776); kvend = 2118452(8473808); length = 13207993/6553600 2016-03-04 20:38:57,209 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 102845638 kvi 25711404(102845616) 2016-03-04 20:39:04,710 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 4 2016-03-04 20:39:04,710 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 102845638 kv 25711404(102845616) kvi 23089976(92359904) 2016-03-04 20:39:06,608 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:39:06,608 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 102845638; bufend = 29043356; bufvoid = 104857592 2016-03-04 20:39:06,608 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 25711404(102845616); kvend = 12503720(50014880); length = 13207685/6553600 2016-03-04 20:39:06,608 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 39529110 kvi 9882272(39529088) 2016-03-04 20:39:13,669 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 5 2016-03-04 20:39:13,669 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 39529110 kv 9882272(39529088) kvi 7260844(29043376) 2016-03-04 20:39:15,551 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:39:15,551 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 39529110; bufend = 70581562; bufvoid = 104857600 2016-03-04 20:39:15,551 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 9882272(39529088); kvend = 22888272(91553088); length = 13208401/6553600 2016-03-04 20:39:15,551 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 81067317 kvi 20266824(81067296) 2016-03-04 20:39:16,351 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2016-03-04 20:39:22,600 INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 6 2016-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 81067317 kv 20266824(81067296) kvi 18022368(72089472) 2016-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output 2016-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 81067317; bufend = 86345178; bufvoid = 104857600 2016-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 20266824(81067296); kvend = 18022372(72089488); length = 2244453/6553600 2016-03-04 20:39:23,552 INFO [main] org.apache.hadoop.mapred.MapTask: Finished spill 7 2016-03-04 20:39:23,565 INFO [main] org.apache.hadoop.mapred.Merger: Merging 8 sorted segments 2016-03-04 20:39:23,577 INFO [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 8 segments left of total size: 1829496 bytes 2016-03-04 20:39:24,096 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1457141718249_0001_m_000000_0 is done. And is in the process of committing 2016-03-04 20:39:24,222 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1457141718249_0001_m_000000_0' done. --001a1143e6900b8a6c052d922b5c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi there,

I run a word count = job in Hadoop 2.6.0 and I see that there are several spills for the=C2=A0 m= ap output.

I have the following configuration:
mapreduce.= task.io.sort.mb =3D 100
mapreduce.map.sort.spill.percent =3D 0.80<= br>

After running the job, the Map output bytes =3D= 222660096.
By looking at the container log below, it seems = that the buffer size is 31055173 bytes not 100MB.

Look at= the parameters values before spilling the first spill into the disk
bu= fstart =3D 0; bufend =3D 31055173; bufvoid =3D 104857600

= And by dividing the Map output bytes by bufend (222660096/31055173 =3D 7.17= (=3D 8 spills)).

Why is this happening? The buffer size= should be 80MB (soft limit), so I think I should have 3 spills only.
Also, can anyone give a better explanation for bufstart, bufend an= d bufvoid?

Thanks,
Sultan
=
=C2=A0

2016-03-04 20:38:16,678 I= NFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hmast= er:9000/input/Input-128M:0+134217728
2016-03-04 20:38:16,842 INFO [main]= org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2= 016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapTask: mapred= uce.task.io.sort.mb: 100
2016-03-04 20:38:16,842 INFO [main] org.apache.= hadoop.mapred.MapTask: soft limit at 83886080
2016-03-04 20:38:16,842 IN= FO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 0; bufvoid =3D 104= 857600
2016-03-04 20:38:16,842 INFO [main] org.apache.hadoop.mapred.MapT= ask: kvstart =3D 26214396; length =3D 6553600
2016-03-04 20:38:16,854 IN= FO [main] org.apache.hadoop.mapred.MapTask: Map output collector class =3D = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-03-04 20:38:19,931= INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output
2016-= 03-04 20:38:19,931 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = =3D 0; bufend =3D 31055173; bufvoid =3D 104857600
2016-03-04 20:38:19,93= 1 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 26214396(104857= 584); kvend =3D 13006672(52026688); length =3D 13207725/6553600
2016-03-= 04 20:38:19,931 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 415= 40922 kvi 10385224(41540896)
2016-03-04 20:38:28,178 INFO [SpillThread] = org.apache.hadoop.mapred.MapTask: Finished spill 0
2016-03-04 20:38:28,1= 79 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 41540922 k= v 10385224(41540896) kvi 7763800(31055200)
2016-03-04 20:38:30,020 INFO = [main] org.apache.hadoop.mapred.MapTask: Spilling map output
2016-03-04 = 20:38:30,020 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 415= 40922; bufend =3D 72594952; bufvoid =3D 104857600
2016-03-04 20:38:30,02= 0 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 10385224(415408= 96); kvend =3D 23391620(93566480); length =3D 13208005/6553600
2016-03-0= 4 20:38:30,020 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 8308= 0708 kvi 20770172(83080688)
2016-03-04 20:38:37,042 INFO [SpillThread] o= rg.apache.hadoop.mapred.MapTask: Finished spill 1
2016-03-04 20:38:37,04= 2 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 83080708 kv= 20770172(83080688) kvi 18148744(72594976)
2016-03-04 20:38:39,045 INFO = [main] org.apache.hadoop.mapred.MapTask: Spilling map output
2016-03-04 = 20:38:39,045 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 830= 80708; bufend =3D 9277592; bufvoid =3D 104857598
2016-03-04 20:38:39,045= INFO [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 20770172(8308068= 8); kvend =3D 7562280(30249120); length =3D 13207893/6553600
2016-03-04 = 20:38:39,045 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 197633= 48 kvi 4940832(19763328)
2016-03-04 20:38:46,376 INFO [SpillThread] org.= apache.hadoop.mapred.MapTask: Finished spill 2
2016-03-04 20:38:46,376 I= NFO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 19763348 kv 49= 40832(19763328) kvi 2319404(9277616)
2016-03-04 20:38:48,320 INFO [main]= org.apache.hadoop.mapred.MapTask: Spilling map output
2016-03-04 20:38:= 48,320 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 19763348;= bufend =3D 50820055; bufvoid =3D 104857600
2016-03-04 20:38:48,320 INFO= [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 4940832(19763328); kv= end =3D 17947892(71791568); length =3D 13207341/6553600
2016-03-04 20:38= :48,320 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 61305803 kv= i 15326444(61305776)
2016-03-04 20:38:55,301 INFO [SpillThread] org.apac= he.hadoop.mapred.MapTask: Finished spill 3
2016-03-04 20:38:55,301 INFO = [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 61305803 kv 153264= 44(61305776) kvi 12705020(50820080)
2016-03-04 20:38:57,209 INFO [main] = org.apache.hadoop.mapred.MapTask: Spilling map output
2016-03-04 20:38:5= 7,209 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 61305803; = bufend =3D 92359884; bufvoid =3D 104857600
2016-03-04 20:38:57,209 INFO = [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 15326444(61305776); kv= end =3D 2118452(8473808); length =3D 13207993/6553600
2016-03-04 20:38:5= 7,209 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 102845638 kvi= 25711404(102845616)
2016-03-04 20:39:04,710 INFO [SpillThread] org.apac= he.hadoop.mapred.MapTask: Finished spill 4
2016-03-04 20:39:04,710 INFO = [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 102845638 kv 25711= 404(102845616) kvi 23089976(92359904)
2016-03-04 20:39:06,608 INFO [main= ] org.apache.hadoop.mapred.MapTask: Spilling map output
2016-03-04 20:39= :06,608 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 10284563= 8; bufend =3D 29043356; bufvoid =3D 104857592
2016-03-04 20:39:06,608 IN= FO [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 25711404(102845616)= ; kvend =3D 12503720(50014880); length =3D 13207685/6553600
2016-03-04 2= 0:39:06,608 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 3952911= 0 kvi 9882272(39529088)
2016-03-04 20:39:13,669 INFO [SpillThread] org.a= pache.hadoop.mapred.MapTask: Finished spill 5
2016-03-04 20:39:13,669 IN= FO [main] org.apache.hadoop.mapred.MapTask: (RESET) equator 39529110 kv 988= 2272(39529088) kvi 7260844(29043376)
2016-03-04 20:39:15,551 INFO [main]= org.apache.hadoop.mapred.MapTask: Spilling map output
2016-03-04 20:39:= 15,551 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart =3D 39529110;= bufend =3D 70581562; bufvoid =3D 104857600
2016-03-04 20:39:15,551 INFO= [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 9882272(39529088); kv= end =3D 22888272(91553088); length =3D 13208401/6553600
2016-03-04 20:39= :15,551 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 81067317 kv= i 20266824(81067296)
2016-03-04 20:39:16,351 INFO [main] org.apache.hado= op.mapred.MapTask: Starting flush of map output
2016-03-04 20:39:22,600 = INFO [SpillThread] org.apache.hadoop.mapred.MapTask: Finished spill 6
20= 16-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: (RESET)= equator 81067317 kv 20266824(81067296) kvi 18022368(72089472)
2016-03-0= 4 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map o= utput
2016-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTa= sk: bufstart =3D 81067317; bufend =3D 86345178; bufvoid =3D 104857600
20= 16-03-04 20:39:22,600 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart= =3D 20266824(81067296); kvend =3D 18022372(72089488); length =3D 2244453/6= 553600
2016-03-04 20:39:23,552 INFO [main] org.apache.hadoop.mapred.MapT= ask: Finished spill 7
2016-03-04 20:39:23,565 INFO [main] org.apache.had= oop.mapred.Merger: Merging 8 sorted segments
2016-03-04 20:39:23,577 INF= O [main] org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with= 8 segments left of total size: 1829496 bytes
2016-03-04 20:39:24,096 IN= FO [main] org.apache.hadoop.mapred.Task: Task:attempt_1457141718249_0001_m_= 000000_0 is done. And is in the process of committing
2016-03-04 20:39:2= 4,222 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_14571417= 18249_0001_m_000000_0' done.
--001a1143e6900b8a6c052d922b5c--