Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 88D81104B8 for ; Sat, 20 Apr 2013 00:39:34 +0000 (UTC) Received: (qmail 84045 invoked by uid 500); 20 Apr 2013 00:39:29 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 83949 invoked by uid 500); 20 Apr 2013 00:39:29 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 83940 invoked by uid 99); 20 Apr 2013 00:39:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Apr 2013 00:39:29 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URIBL_DBL_REDIR X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of geelongyao@gmail.com designates 209.85.128.51 as permitted sender) Received: from [209.85.128.51] (HELO mail-qe0-f51.google.com) (209.85.128.51) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Apr 2013 00:39:24 +0000 Received: by mail-qe0-f51.google.com with SMTP id 1so3153431qec.10 for ; Fri, 19 Apr 2013 17:39:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:x-mailer:date:message-id:in-reply-to :references:x-orchestra-oid:x-orchestra-sig:x-orchestra-thrid :x-orchestra-thrid-sig:x-orchestra-account:from:to:subject :content-type; bh=xBBIjUNIiYIqQ9JUddcKc6Ec2n8e98sDxDojh0UQfzY=; b=LeYtW5Gsu8oPVgE/hU9jx4Zy8otsGuU2FL8i0YahbSu8dqGI+tb44QsvLrJ7EQB5F8 q8SLwndtMB58AXV5rsUM9L259/u4vOsjc+wI78Hmm8n5GgpD5mr6E1pYKcnee5XPRDRt wZXBXA/Q7GMQLJKXgNQLmjhJ9CiYH1n2SCIxxJ3f4YAnSQHUdxzP+xqUg65dTu/cLqeP d1Ful14fdSGYrYPIbJBUfggXaAMcoHooU3ysVNGtQMeFh0QkPNbmHZ36pury+dsXe1a9 zY6nxNBCJXQOWYWMu2bBEAD33kV41HFVNe6l+AcllkcQpleiYDA9hwX4ZSxIFt/2UuO3 ybAw== X-Received: by 10.229.161.145 with SMTP id r17mr2304459qcx.93.1366418343472; Fri, 19 Apr 2013 17:39:03 -0700 (PDT) Received: from [127.0.0.1] (mail-2.mailboxapp.com. [54.235.133.3]) by mx.google.com with ESMTPS id ed8sm19894361qeb.7.2013.04.19.17.39.02 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 19 Apr 2013 17:39:02 -0700 (PDT) MIME-Version: 1.0 X-Mailer: Nodemailer (0.3.28; +http://andris9.github.com/Nodemailer/) Date: Fri, 19 Apr 2013 17:39:02 -0700 (PDT) Message-Id: <1366418342477.e8cd4cf6@Nodemailer> In-Reply-To: References: X-Orchestra-Oid: 593FF874-6BAA-46B5-AE02-F18D2E2E6C9C X-Orchestra-Sig: 0fbee30940b01c8608c0f7d8cb180cfd2fe08d57 X-Orchestra-Thrid: T71E25CF3-6EA0-4367-824C-0FD1BBD8DD45_1432762747841242941 X-Orchestra-Thrid-Sig: fe687bae2f141036b9121df1ffb8d91180fb0242 X-Orchestra-Account: e01b49c63b7efe85b979e2db57e8a14e8a21fd59 From: "=?UTF-8?Q?=E5=A7=9A=E5=90=89=E9=BE=99?=" To: user@hadoop.apache.org Subject: Re: =?UTF-8?Q?Map=E2=80=98s?= number with NLineInputFormat Content-Type: multipart/alternative; boundary="----mailcomposer-?=_1-1366418342979" X-Virus-Checked: Checked by ClamAV on apache.org ------mailcomposer-?=_1-1366418342979 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable The num of map is decided by the block size and your rawdata=C2=A0 =E2=80=94 Sent from Mailbox for iPhone On Sat, Apr 20, 2013 at 12:30 AM, YouPeng Yang wrote: > Hi All > I take NLineInputFormat as the Text Input Format with the following = code > : > NLineInputFormat.setNumLinesPerSplit(job, 10); > NLineInputFormat.addInputPath(job,new Path(args[0].toString())); > My input file contains 1000 rows,so I thought it will distribute > 100(1000/10) maps.However I got 4 maps. > I'm confued by the number of Map that was distributed according to the > running log[1]. > How it distribute maps when using NLineInputFormat > Regards > [1]=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > .... > .... > 2013-04-19 23:56:20,377 INFO mapreduce.Job > (Job.java:monitorAndPrintJob(1286)) - Job job=5Flocal=5F0001 running in = uber > mode : false > 2013-04-19 23:56:20,377 INFO mapreduce.Job > (Job.java:monitorAndPrintJob(1293)) - map 25% reduce 0% > 2013-04-19 23:56:20,381 INFO mapred.MapTask > (MapTask.java:sortAndSpill(1597)) - Finished spill 0 > 2013-04-19 23:56:20,384 INFO mapred.Task (Task.java:done(979)) - > Task:attempt=5Flocal=5F0001=5Fm=5F000001=5F0 is done. And is in the = process of > committing > 2013-04-19 23:56:20,388 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - map > 2013-04-19 23:56:20,389 INFO mapred.Task (Task.java:sendDone(1099)) - = Task > 'attempt=5Flocal=5F0001=5Fm=5F000001=5F0' done. > 2013-04-19 23:56:20,389 INFO mapred.LocalJobRunner > (LocalJobRunner.java:run(238)) - Finishing task: > attempt=5Flocal=5F0001=5Fm=5F000001=5F0 > 2013-04-19 23:56:20,389 INFO mapred.LocalJobRunner > (LocalJobRunner.java:run(213)) - Starting task: > attempt=5Flocal=5F0001=5Fm=5F000002=5F0 > 2013-04-19 23:56:20,391 INFO mapred.Task (Task.java:initialize(565)) - > Using ResourceCalculatorPlugin : > org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@36bf7916 > 2013-04-19 23:56:20,486 INFO mapred.MapTask > (MapTask.java:setEquator(1127)) - (EQUATOR) 0 kvi 26214396(104857584) > 2013-04-19 23:56:20,486 INFO mapred.MapTask (MapTask.java:(923)) = - > mapreduce.task.io.sort.mb: 100 > 2013-04-19 23:56:20,486 INFO mapred.MapTask (MapTask.java:(924)) = - > soft limit at 83886080 > 2013-04-19 23:56:20,486 INFO mapred.MapTask (MapTask.java:(925)) = - > bufstart =3D 0; bufvoid =3D 104857600 > 2013-04-19 23:56:20,487 INFO mapred.MapTask (MapTask.java:(926)) = - > kvstart =3D 26214396; length =3D 6553600 > 2013-04-19 23:56:20,515 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - > 2013-04-19 23:56:20,515 INFO mapred.MapTask (MapTask.java:flush(1389)) = - > Starting flush of map output > 2013-04-19 23:56:20,516 INFO mapred.MapTask (MapTask.java:flush(1408)) = - > Spilling map output > 2013-04-19 23:56:20,516 INFO mapred.MapTask (MapTask.java:flush(1409)) = - > bufstart =3D 0; bufend =3D 336; bufvoid =3D 104857600 > 2013-04-19 23:56:20,516 INFO mapred.MapTask (MapTask.java:flush(1411)) = - > kvstart =3D 26214396(104857584); kvend =3D 26214208(104856832); length = =3D > 189/6553600 > 2013-04-19 23:56:20,523 INFO mapred.MapTask > (MapTask.java:sortAndSpill(1597)) - Finished spill 0 > 2013-04-19 23:56:20,552 INFO mapred.Task (Task.java:done(979)) - > Task:attempt=5Flocal=5F0001=5Fm=5F000002=5F0 is done. And is in the = process of > committing > 2013-04-19 23:56:20,555 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - map > 2013-04-19 23:56:20,556 INFO mapred.Task (Task.java:sendDone(1099)) - = Task > 'attempt=5Flocal=5F0001=5Fm=5F000002=5F0' done. > 2013-04-19 23:56:20,556 INFO mapred.LocalJobRunner > (LocalJobRunner.java:run(238)) - Finishing task: > attempt=5Flocal=5F0001=5Fm=5F000002=5F0 > 2013-04-19 23:56:20,556 INFO mapred.LocalJobRunner > (LocalJobRunner.java:run(213)) - Starting task: > attempt=5Flocal=5F0001=5Fm=5F000003=5F0 > 2013-04-19 23:56:20,558 INFO mapred.Task (Task.java:initialize(565)) - > Using ResourceCalculatorPlugin : > org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@746a63d3 > 2013-04-19 23:56:20,666 INFO mapred.MapTask > (MapTask.java:setEquator(1127)) - (EQUATOR) 0 kvi 26214396(104857584) > 2013-04-19 23:56:20,666 INFO mapred.MapTask (MapTask.java:(923)) = - > mapreduce.task.io.sort.mb: 100 > 2013-04-19 23:56:20,666 INFO mapred.MapTask (MapTask.java:(924)) = - > soft limit at 83886080 > 2013-04-19 23:56:20,666 INFO mapred.MapTask (MapTask.java:(925)) = - > bufstart =3D 0; bufvoid =3D 104857600 > 2013-04-19 23:56:20,667 INFO mapred.MapTask (MapTask.java:(926)) = - > kvstart =3D 26214396; length =3D 6553600 > 2013-04-19 23:56:20,690 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - > 2013-04-19 23:56:20,690 INFO mapred.MapTask (MapTask.java:flush(1389)) = - > Starting flush of map output > 2013-04-19 23:56:20,690 INFO mapred.MapTask (MapTask.java:flush(1408)) = - > Spilling map output > 2013-04-19 23:56:20,690 INFO mapred.MapTask (MapTask.java:flush(1409)) = - > bufstart =3D 0; bufend =3D 329; bufvoid =3D 104857600 > 2013-04-19 23:56:20,690 INFO mapred.MapTask (MapTask.java:flush(1411)) = - > kvstart =3D 26214396(104857584); kvend =3D 26214212(104856848); length = =3D > 185/6553600 > 2013-04-19 23:56:20,695 INFO mapred.MapTask > (MapTask.java:sortAndSpill(1597)) - Finished spill 0 > 2013-04-19 23:56:20,697 INFO mapred.Task (Task.java:done(979)) - > Task:attempt=5Flocal=5F0001=5Fm=5F000003=5F0 is done. And is in the = process of > committing > 2013-04-19 23:56:20,717 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - map > 2013-04-19 23:56:20,718 INFO mapred.Task (Task.java:sendDone(1099)) - = Task > 'attempt=5Flocal=5F0001=5Fm=5F000003=5F0' done. > 2013-04-19 23:56:20,718 INFO mapred.LocalJobRunner > (LocalJobRunner.java:run(238)) - Finishing task: > attempt=5Flocal=5F0001=5Fm=5F000003=5F0 > 2013-04-19 23:56:20,718 INFO mapred.LocalJobRunner > (LocalJobRunner.java:run(394)) - Map task executor complete. > 2013-04-19 23:56:20,752 INFO mapred.Task (Task.java:initialize(565)) - > Using ResourceCalculatorPlugin : > org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@52cd19d > 2013-04-19 23:56:20,760 INFO mapred.Merger (Merger.java:merge(549)) - > Merging 4 sorted segments > 2013-04-19 23:56:20,767 INFO mapred.Merger (Merger.java:merge(648)) - = Down > to the last merge-pass, with 4 segments left of total size: 8532 bytes > 2013-04-19 23:56:20,768 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - > 2013-04-19 23:56:20,807 WARN conf.Configuration > (Configuration.java:warnOnceIfDeprecated(808)) - mapred.skip.on is > deprecated. Instead, use mapreduce.job.skiprecords > 2013-04-19 23:56:21,129 INFO mapred.Task (Task.java:done(979)) - > Task:attempt=5Flocal=5F0001=5Fr=5F000000=5F0 is done. And is in the = process of > committing > 2013-04-19 23:56:21,131 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - > 2013-04-19 23:56:21,131 INFO mapred.Task (Task.java:commit(1140)) - = Task > attempt=5Flocal=5F0001=5Fr=5F000000=5F0 is allowed to commit now > 2013-04-19 23:56:21,138 INFO output.FileOutputCommitter > (FileOutputCommitter.java:commitTask(432)) - Saved output of task > 'attempt=5Flocal=5F0001=5Fr=5F000000=5F0' to > hdfs://Hadoop01:8040/user/hadoop/d/multi9/=5Ftemporary/0/task=5Flocal=5F0= 001=5Fr=5F000000 > 2013-04-19 23:56:21,139 INFO mapred.LocalJobRunner > (LocalJobRunner.java:statusUpdate(501)) - reduce > reduce > 2013-04-19 23:56:21,139 INFO mapred.Task (Task.java:sendDone(1099)) - = Task > 'attempt=5Flocal=5F0001=5Fr=5F000000=5F0' done. > 2013-04-19 23:56:21,381 INFO mapreduce.Job > (Job.java:monitorAndPrintJob(1293)) - map 100% reduce 100% > 2013-04-19 23:56:21,381 INFO mapreduce.Job > (Job.java:monitorAndPrintJob(1304)) - Job job=5Flocal=5F0001 completed > successfully > 2013-04-19 23:56:21,427 INFO mapreduce.Job > (Job.java:monitorAndPrintJob(1311)) - Counters: 32 > File System Counters > FILE: Number of bytes read=3D483553 > FILE: Number of bytes written=3D1313962 > FILE: Number of read operations=3D0 > FILE: Number of large read operations=3D0 > FILE: Number of write operations=3D0 > HDFS: Number of bytes read=3D296769 > HDFS: Number of bytes written=3D284 > HDFS: Number of read operations=3D66 > HDFS: Number of large read operations=3D0 > HDFS: Number of write operations=3D8 > Map-Reduce Framework > Map input records=3D1000 > Map output records=3D1000 > Map output bytes=3D6543 > Map output materialized bytes=3D8567 > Input split bytes=3D516 > Combine input records=3D0 > Combine output records=3D0 > Reduce input groups=3D12 > Reduce shuffle bytes=3D0 > Reduce input records=3D1000 > Reduce output records=3D0 > Spilled Records=3D2000 > Shuffled Maps =3D0 > Failed Shuffles=3D0 > Merged Map outputs=3D0 > GC time elapsed (ms)=3D7 > CPU time spent (ms)=3D0 > Physical memory (bytes) snapshot=3D0 > Virtual memory (bytes) snapshot=3D0 > Total committed heap usage (bytes)=3D1773993984 > File Input Format Counters > Bytes Read=3D68723 > File Output Format Counters > Bytes Written=3D0 ------mailcomposer-?=_1-1366418342979 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

The num of map is decided by the block size and your = rawdata=C2=A0

=E2=80=94
Sent = from Mailbox for = iPhone


On Sat, Apr 20, = 2013 at 12:30 AM, YouPeng Yang <yypvs= xf19870706@gmail.com> wrote:

Hi All
=C2=A0 = =C2=A0
=C2=A0I =C2=A0take NLineInputFormat = =C2=A0as the Text Input Format with the following code :
=C2=A0NLineI= nputFormat.setNumLinesPerSplit(job, 10);
=C2=A0NLineInputFormat.= addInputPath(job,new Path(args[0].toString()));

= =C2=A0My input file contains 1000 rows,so I thought it will distribute = 100(1000/10) maps.However I got 4 maps.

=C2=A0 I'm confued by the = number of Map that was distributed according to the running log[1].=
=C2=A0How it distribute =C2=A0maps when = using=C2=A0NLineInputFormat


Regards



[1]=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
....
....
2013-04-19 23:56:20,377 INFO =C2=A0mapreduce.Job (Job.= java:monitorAndPrintJob(1286)) - Job job=5Flocal=5F0001 running in uber = mode : false
2013-04-19 23:56:20,377 INFO =C2=A0mapreduce.Job = (Job.java:monitorAndPrintJob(1293)) - =C2=A0map 25% reduce 0%
2013-04-19 23:56:20,381 INFO =C2=A0mapred.MapTask (MapTask.= java:sortAndSpill(1597)) - Finished spill 0
2013-04-19 23:56:20,= 384 INFO =C2=A0mapred.Task (Task.java:done(979)) - Task:attempt=5Flocal=5F0= 001=5Fm=5F000001=5F0 is done. And is in the process of committing
2013-04-19 23:56:20,388 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:statusUpdate(501)) - map
2013-04-19 23:56:20= ,389 INFO =C2=A0mapred.Task (Task.java:sendDone(1099)) - Task = 'attempt=5Flocal=5F0001=5Fm=5F000001=5F0' done.
2013-04-19 23:56:20,389 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:run(238)) - Finishing task: attempt=5Flocal=5F0001=5Fm= =5F000001=5F0
2013-04-19 23:56:20,389 INFO =C2=A0mapred.= LocalJobRunner (LocalJobRunner.java:run(213)) - Starting task: = attempt=5Flocal=5F0001=5Fm=5F000002=5F0
2013-04-19 23:56:20,391 INFO =C2=A0mapred.Task (Task.= java:initialize(565)) - =C2=A0Using ResourceCalculatorPlugin : org.apache.= hadoop.yarn.util.LinuxResourceCalculatorPlugin@36bf7916
2013-04-1= 9 23:56:20,486 INFO =C2=A0mapred.MapTask (MapTask.java:setEquator(1127)) - = (EQUATOR) 0 kvi 26214396(104857584)
2013-04-19 23:56:20,486 INFO =C2=A0mapred.MapTask (MapTask.= java:<init>(923)) - mapreduce.task.io.sort.mb: = 100
2013-04-19 23:56:20,486 INFO =C2=A0mapred.MapTask (MapTask.= java:<init>(924)) - soft limit at 83886080
2013-04-19 23:56:20,486 INFO =C2=A0mapred.MapTask (MapTask.= java:<init>(925)) - bufstart =3D 0; bufvoid =3D = 104857600
2013-04-19 23:56:20,487 INFO =C2=A0mapred.MapTask = (MapTask.java:<init>(926)) - kvstart =3D 26214396; length =3D = 6553600
2013-04-19 23:56:20,515 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:statusUpdate(501)) -=C2=A0
2013-04-19 = 23:56:20,515 INFO =C2=A0mapred.MapTask (MapTask.java:flush(1389)) - = Starting flush of map output
2013-04-19 23:56:20,516 INFO =C2=A0mapred.MapTask (MapTask.= java:flush(1408)) - Spilling map output
2013-04-19 23:56:20,516 = INFO =C2=A0mapred.MapTask (MapTask.java:flush(1409)) - bufstart =3D 0; = bufend =3D 336; bufvoid =3D 104857600
2013-04-19 23:56:20,516 INFO =C2=A0mapred.MapTask (MapTask.= java:flush(1411)) - kvstart =3D 26214396(104857584); kvend =3D = 26214208(104856832); length =3D 189/6553600
2013-04-19 23:56:20,= 523 INFO =C2=A0mapred.MapTask (MapTask.java:sortAndSpill(1597)) - Finished = spill 0
2013-04-19 23:56:20,552 INFO =C2=A0mapred.Task (Task.java:done(979)) -= Task:attempt=5Flocal=5F0001=5Fm=5F000002=5F0 is done. And is in the = process of committing
2013-04-19 23:56:20,555 INFO =C2=A0mapred.= LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) - map
2013-04-19 23:56:20,556 INFO =C2=A0mapred.Task (Task.= java:sendDone(1099)) - Task 'attempt=5Flocal=5F0001=5Fm=5F000002=5F0' done.=
2013-04-19 23:56:20,556 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:run(238)) - Finishing task: attempt=5Flocal=5F0001=5Fm= =5F000002=5F0
2013-04-19 23:56:20,556 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:run(213)) - Starting task: attempt=5Flocal=5F0001=5Fm= =5F000003=5F0
2013-04-19 23:56:20,558 INFO =C2=A0mapred.Task = (Task.java:initialize(565)) - =C2=A0Using ResourceCalculatorPlugin : org.= apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@746a63d3
2013-04-19 23:56:20,666 INFO =C2=A0mapred.MapTask (MapTask.= java:setEquator(1127)) - (EQUATOR) 0 kvi 26214396(104857584)
2013= -04-19 23:56:20,666 INFO =C2=A0mapred.MapTask (MapTask.= java:<init>(923)) - mapreduce.task.io.sort.mb: 100
2013-04-19 23:56:20,666 INFO =C2=A0mapred.MapTask (MapTask.= java:<init>(924)) - soft limit at 83886080
2013-04-19 = 23:56:20,666 INFO =C2=A0mapred.MapTask (MapTask.java:<init>(925)) - = bufstart =3D 0; bufvoid =3D 104857600
2013-04-19 23:56:20,667 INFO =C2=A0mapred.MapTask (MapTask.= java:<init>(926)) - kvstart =3D 26214396; length =3D = 6553600
2013-04-19 23:56:20,690 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:statusUpdate(501)) -=C2=A0
2013-04-19 23:56:20,690 INFO =C2=A0mapred.MapTask (MapTask.= java:flush(1389)) - Starting flush of map output
2013-04-19 = 23:56:20,690 INFO =C2=A0mapred.MapTask (MapTask.java:flush(1408)) - = Spilling map output
2013-04-19 23:56:20,690 INFO =C2=A0mapred.MapTask (MapTask.= java:flush(1409)) - bufstart =3D 0; bufend =3D 329; bufvoid =3D = 104857600
2013-04-19 23:56:20,690 INFO =C2=A0mapred.MapTask = (MapTask.java:flush(1411)) - kvstart =3D 26214396(104857584); kvend =3D = 26214212(104856848); length =3D 185/6553600
2013-04-19 23:56:20,695 INFO =C2=A0mapred.MapTask (MapTask.= java:sortAndSpill(1597)) - Finished spill 0
2013-04-19 23:56:20,= 697 INFO =C2=A0mapred.Task (Task.java:done(979)) - Task:attempt=5Flocal=5F0= 001=5Fm=5F000003=5F0 is done. And is in the process of committing
2013-04-19 23:56:20,717 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:statusUpdate(501)) - map
2013-04-19 23:56:20= ,718 INFO =C2=A0mapred.Task (Task.java:sendDone(1099)) - Task = 'attempt=5Flocal=5F0001=5Fm=5F000003=5F0' done.
2013-04-19 23:56:20,718 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:run(238)) - Finishing task: attempt=5Flocal=5F0001=5Fm= =5F000003=5F0
2013-04-19 23:56:20,718 INFO =C2=A0mapred.= LocalJobRunner (LocalJobRunner.java:run(394)) - Map task executor complete.=
2013-04-19 23:56:20,752 INFO =C2=A0mapred.Task (Task.= java:initialize(565)) - =C2=A0Using ResourceCalculatorPlugin : org.apache.= hadoop.yarn.util.LinuxResourceCalculatorPlugin@52cd19d
2013-04-19= 23:56:20,760 INFO =C2=A0mapred.Merger (Merger.java:merge(549)) - Merging 4= sorted segments
2013-04-19 23:56:20,767 INFO =C2=A0mapred.Merger (Merger.= java:merge(648)) - Down to the last merge-pass, with 4 segments left of = total size: 8532 bytes
2013-04-19 23:56:20,768 INFO =C2=A0mapred.= LocalJobRunner (LocalJobRunner.java:statusUpdate(501)) -=C2=A0
2013-04-19 23:56:20,807 WARN =C2=A0conf.Configuration (Configuration.= java:warnOnceIfDeprecated(808)) - mapred.skip.on is deprecated. Instead, = use mapreduce.job.skiprecords
2013-04-19 23:56:21,129 INFO = =C2=A0mapred.Task (Task.java:done(979)) - Task:attempt=5Flocal=5F0001=5Fr= =5F000000=5F0 is done. And is in the process of committing
2013-04-19 23:56:21,131 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:statusUpdate(501)) -=C2=A0
2013-04-19 = 23:56:21,131 INFO =C2=A0mapred.Task (Task.java:commit(1140)) - Task = attempt=5Flocal=5F0001=5Fr=5F000000=5F0 is allowed to commit now
2013-04-19 23:56:21,138 INFO =C2=A0output.FileOutputCommitter = (FileOutputCommitter.java:commitTask(432)) - Saved output of task = 'attempt=5Flocal=5F0001=5Fr=5F000000=5F0' to hdfs://Hadoop01:8040/user/hado= op/d/multi9/=5Ftemporary/0/task=5Flocal=5F0001=5Fr=5F000000
2013-04-19 23:56:21,139 INFO =C2=A0mapred.LocalJobRunner = (LocalJobRunner.java:statusUpdate(501)) - reduce > = reduce
2013-04-19 23:56:21,139 INFO =C2=A0mapred.Task (Task.= java:sendDone(1099)) - Task 'attempt=5Flocal=5F0001=5Fr=5F000000=5F0' done.=
2013-04-19 23:56:21,381 INFO =C2=A0mapreduce.Job (Job.= java:monitorAndPrintJob(1293)) - =C2=A0map 100% reduce = 100%
2013-04-19 23:56:21,381 INFO =C2=A0mapreduce.Job (Job.= java:monitorAndPrintJob(1304)) - Job job=5Flocal=5F0001 completed = successfully
2013-04-19 23:56:21,427 INFO =C2=A0mapreduce.Job (Job.= java:monitorAndPrintJob(1311)) - Counters: 32
File System = Counters
= FILE: Number of bytes read=3D483553
FILE: = Number of bytes written=3D1313962
FILE: Number of read = operations=3D0
FILE: Number of large read operations=3D0
FILE: = Number of write operations=3D0
HDFS: Number of bytes = read=3D296769
= HDFS: Number of bytes written=3D284
HDFS: = Number of read operations=3D66
HDFS: Number of large read = operations=3D0
HDFS: Number of write operations=3D8
Map-Reduce = Framework
= Map input records=3D1000
Map output records=3D1000
Map output= bytes=3D6543
= Map output materialized bytes=3D8567
Input split = bytes=3D516
Combine = input records=3D0
Combine output records=3D0
Reduce input groups=3D12
Reduce = shuffle bytes=3D0
Reduce input records=3D1000
Reduce output records=3D0
Spilled = Records=3D2000
Shuffled Maps =3D0
Failed Shuffles=3D0
Merged Map= outputs=3D0
= GC time elapsed (ms)=3D7
CPU time spent (ms)=3D0
Physical = memory (bytes) snapshot=3D0
Virtual memory (bytes) = snapshot=3D0
= Total committed heap usage (bytes)=3D1773993984
File Input = Format Counters=C2=A0
Bytes Read=3D68723
File Output Format = Counters=C2=A0
Bytes = Written=3D0




------mailcomposer-?=_1-1366418342979--