Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 609FB11C62 for ; Fri, 21 Feb 2014 05:37:30 +0000 (UTC) Received: (qmail 58630 invoked by uid 500); 21 Feb 2014 05:37:23 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 58536 invoked by uid 500); 21 Feb 2014 05:37:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 58529 invoked by uid 99); 21 Feb 2014 05:37:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Feb 2014 05:37:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vinodkv@hortonworks.com designates 209.85.160.51 as permitted sender) Received: from [209.85.160.51] (HELO mail-pb0-f51.google.com) (209.85.160.51) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Feb 2014 05:37:14 +0000 Received: by mail-pb0-f51.google.com with SMTP id un15so2953639pbc.24 for ; Thu, 20 Feb 2014 21:36:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:message-id:mime-version:subject:date :references:to:in-reply-to:content-type; bh=xynsNltfDcWzVoIvuJeMdBN90Z0Q0SuZHM+GbYcsBI8=; b=ZqPxyXvnbxrBkSc+ZxmQviCYkIAaN6/v64u3NIqtubPDjy7kZv/7nOXdSXw+YyMgFN m9F9nN53P2zO7hMsK++fDnR8d1MqIAEjeBxvE2Elq9otZDW3ce2PnSEE51CEUydOZj1Y ZGjLU0K5CczvC18cxlUxXzbBJJwzi4vua2ifSPTL0wrbiX+0eA5NiPphZm8pAf12uf8S +DVpmKlwPdRpXQGMYGQrQYEQIVPKX9jbL47Jb5Jr0nP3nvdz6TxAnWro62U+t6fukhWk 6aNS0hqNnB8JgBl3bMNhNGAs6fiXvcEczHRDXA1akqbPZrJkl2ZSxkkJGYCVvEf2vqTg gyYA== X-Gm-Message-State: ALoCoQnz2CxjS4usy2afu8wHYdtlJpMNIwbSj5idfBLNbGrFUfn9Lq7XKArIcafUz4W5Kt9p1Kyj00TuCzTbZTNNnfeNn8+JeU+792m776sQsUeYi9rhHxw= X-Received: by 10.67.22.100 with SMTP id hr4mr6914977pad.112.1392961012680; Thu, 20 Feb 2014 21:36:52 -0800 (PST) Received: from [10.11.2.216] ([192.175.27.2]) by mx.google.com with ESMTPSA id lh13sm41304331pab.4.2014.02.20.21.36.45 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 20 Feb 2014 21:36:45 -0800 (PST) Sender: Vinod Kumar Vavilapalli From: Vinod Kumar Vavilapalli Message-Id: <117F8533-04E8-4227-A2AF-901339899D71@apache.org> Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example Date: Thu, 20 Feb 2014 21:36:43 -0800 References: <58451DF3D30A204A802C12E039768BEFF40583@SHSMSX102.ccr.corp.intel.com> To: user@hadoop.apache.org In-Reply-To: <58451DF3D30A204A802C12E039768BEFF40583@SHSMSX102.ccr.corp.intel.com> X-Mailer: Apple Mail (2.1827) Content-Type: multipart/mixed; boundary=047d7b5dadd05ad22004f2e402df X-Virus-Checked: Checked by ClamAV on apache.org --047d7b5dadd05ad22004f2e402df Content-Type: multipart/alternative; boundary="Apple-Mail=_C9A6C2EC-78F2-4D83-BD80-0997929628FB" --Apple-Mail=_C9A6C2EC-78F2-4D83-BD80-0997929628FB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Are you trying to run it on a cluster or locally? There is something called= 'local mode' which is getting activated here (see that job-id has a _local= _ string). You need to point your configuration to the real ResourceManager. Share you= r configs. Or better share the resource that you are following as instructi= ons. +Viond On Feb 19, 2014, at 10:09 PM, Chen, Richard wrote: > Dear Hadoop User group, > =20 > This is my virgin post. So please bear with me if I made any mistakes at = my early time here. > =20 > Recently I successfully downloaded and compiled Hadoop trunk code (svn co= http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk) > I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical clus= ter of 5 nodes each running 64-bit Cent OS 6. > =20 > After careful configuration, when I tried running the 1st PI job example: > $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPS= HOT.jar pi 4 10 > =20 > The result was successfully calculated but some exception message was als= o dumped during the execution and I couldn=92t figure out why. > The exception message goes like this: > =20 > Starting Job > 14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecate= d. Instead, use dfs.metrics.session-id > 14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with proc= essName=3DJobTracker, sessionId=3D > 14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to proces= s : 4 > 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4 > 14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job:= job_local1598491292_0001 > 14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://lo= calhost:8080/ > 14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_00= 01 > 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in conf= ig null > 14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apac= he.hadoop.mapreduce.lib.output.FileOutputCommitter > 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks > 14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_loca= l1598491292_0001_m_000000_0 > 14/02/20 13:34:13 INFO mapred.Task: Using ResourceCalculatorProcessTree = : [ ] > 14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9= 000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118 > 14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class =3D org= .apache.hadoop.mapred.MapTask$MapOutputBuffer > 14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584= ) > 14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 > 14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080 > 14/02/20 13:34:13 INFO mapred.MapTask: bufstart =3D 0; bufvoid =3D 104857= 600 > 14/02/20 13:34:13 INFO mapred.MapTask: kvstart =3D 26214396; length =3D 6= 553600 > 14/02/20 13:34:13 INFO mapred.LocalJobRunner: > 14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output > 14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output > 14/02/20 13:34:13 INFO mapred.MapTask: bufstart =3D 0; bufend =3D 18; buf= void =3D 104857600 > 14/02/20 13:34:13 INFO mapred.MapTask: kvstart =3D 26214396(104857584); k= vend =3D 26214392(104857568); length =3D 5/6553600 > 14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0 > 14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_0= 00000_0 is done. And is in the process of committing > 14/02/20 13:34:13 WARN mapred.Task: Could not find output size > java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/lo= cal/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local15984= 91292_0001_m_000000_0/output/file.out > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall= (DistributedFileSystem.java:1072) > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall= (DistributedFileSystem.java:1064) > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(Fi= leSystemLinkResolver.java:81) > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileSt= atus(DistributedFileSystem.java:1064) > at org.apache.hadoop.mapred.Task.calculateOutputSize(Task= .java:1124) > at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java= :1102) > at org.apache.hadoop.mapred.Task.done(Task.java:1048) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344) > at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRun= nable.run(LocalJobRunner.java:244) > at java.util.concurrent.Executors$RunnableAdapter.call(Ex= ecutors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:26= 2) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Thre= adPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thr= eadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > =20 > =20 > Please kindly help me on this. > =20 > Thanks in advance! > =20 > Richard --=20 CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to= =20 which it is addressed and may contain information that is confidential,=20 privileged and exempt from disclosure under applicable law. If the reader= =20 of this message is not the intended recipient, you are hereby notified that= =20 any printing, copying, dissemination, distribution, disclosure or=20 forwarding of this communication is strictly prohibited. If you have=20 received this communication in error, please contact the sender immediately= =20 and delete it from your system. Thank You. --Apple-Mail=_C9A6C2EC-78F2-4D83-BD80-0997929628FB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Are you trying to run = it on a cluster or locally? There is something called 'local mode' which is= getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. S= hare your configs. Or better share the resource that you are following as i= nstructions.

+Viond

On Feb= 19, 2014, at 10:09 PM, Chen, Richard <richard.chen@intel.com> wrote:

Dear Hadoop User group,
 
This is my virg= in post. So please bear with me if I made any mistakes at my early time her= e.
 
Recently = I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hado= op-trunk)
I also deployed the hadoop-3.0.0= -SNAPSHOT distribution to a physical cluster of 5 nodes each running 64-bit= Cent OS 6.
 
After careful configuration, when I tried running the 1st PI job example:<= /o:p>
 $hadoop jar share/hadoop/mapreduce/hadoop= -mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10
 
The result was successfully = calculated but some exception message was also dumped during the execution = and I couldn=92t figure out why.
The excep= tion message goes like this:
 
Starting Job
14= /02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. I= nstead, use dfs.metrics.session-id
14/02/2= 0 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName= =3DJobTracker, sessionId=3D
14/02/20 13:34= :12 INFO input.FileInputFormat: Total input paths to process : 4=
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: numbe= r of splits:4
14/02/20 13:34:12 INFO mapre= duce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001=
14/02/20 13:34:12 INFO mapreduce.Job: The url = to track the job: http://localhost:8080/
14/02/20 = 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: Outp= utCommitter set in config null
14/02/20 13= :34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.map= reduce.lib.output.FileOutputCommitter
14/0= 2/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks<= /div>
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Startin= g task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorPr= ocessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class= =3D org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396= (104857584)
14/02/20 13:34:13 INFO mapred.= MapTask: mapreduce.task.io.sort.mb: 100
14= /02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart =3D 0; bu= fvoid =3D 104857600
14/02/20 13:34:13 INFO= mapred.MapTask: kvstart =3D 26214396; length =3D 6553600
<= div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibr= i, sans-serif;">14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush o= f map output
14/02/20 13:34:13 INFO mapred= .MapTask: Spilling map output
14/02/20 13:= 34:13 INFO mapred.MapTask: bufstart =3D 0; bufend =3D 18; bufvoid =3D 10485= 7600
14/02/20 13:34:13 INFO mapred.MapTask= : kvstart =3D 26214396(104857584); kvend =3D 26214392(104857568); length = =3D 5/6553600
14/02/20 13:34:13 INFO mapre= d.MapTask: Finished spill 0
14/02/20 13:34= :13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done.= And is in the process of committing
14/02= /20 13:34:13 WARN mapred.Task: Could not find output size
<= div style=3D"margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibr= i, sans-serif;">java.io.FileNotFoundException: File does not exist: /tmp/ha= doop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/atte= mpt_local1598491292_0001_m_000000_0/output/file.out
           = ;     at org.apache.hadoop.hdfs.DistributedFileSystem$1= 7.doCall(DistributedFileSystem.java:1072)
=             &nb= sp;   at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(D= istributedFileSystem.java:1064)
 &nbs= p;            &= nbsp; at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLink= Resolver.java:81)
    =             at org.a= pache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem= .java:1064)
     =            at org.apache.= hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
          &n= bsp;     at org.apache.hadoop.mapred.Task.sendLastUpdat= e(Task.java:1102)
    =             at org.a= pache.hadoop.mapred.Task.done(Task.java:1048)
            = ;    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:34= 4)
      &nb= sp;         at org.apache.hadoop.ma= pred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
        =         at java.util.concurrent.Executor= s$RunnableAdapter.call(Executors.java:471)
            &n= bsp;   at java.util.concurrent.FutureTask.run(FutureTask.java:262= )
      &nbs= p;         at java.util.concurrent.= ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
=
         &nbs= p;      at java.util.concurrent.ThreadPoolExecutor= $Worker.run(ThreadPoolExecutor.java:615)
&= nbsp;           &nbs= p;   at java.lang.Thread.run(Thread.java:744)
 
 
Please kindly help = me on this.
 
Thanks = in advance!
 
Richard


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --Apple-Mail=_C9A6C2EC-78F2-4D83-BD80-0997929628FB-- --047d7b5dadd05ad22004f2e402df Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJTBuXrAAoJECVa31bDbF8PuKUH/jq8eO8rIk+qS0KQC8vmw/rx nFhTQ8KHfhxjTj5/JqwDpLwD9v0VqEPncyiOvg4IjVVTSirmkBbDRpNAwTgY3RCv KWCtVYLWuJygZ8FqoQol9Mav02m8WpKfJVAtqUIUd0geOrlO+ilEksUYYlk7lUSt EZk1LxhIAsDe9u8CkKcXPAhwwN11EesEelrj2id66poZrThslz7bND66tKu+1noM xKuqhLLWJSWSmmD9ZuB0Z9vjlLoPTC7xd7bITcAFA51pDF8DhWYkqy6VAhC08tkk DVu8x2H22+B8rWUTN1CQivZhNOIwLLKu7xYvR6NEbd5qva1PVYLdVU2UiHtOHA0= =Fqu4 -----END PGP SIGNATURE----- --047d7b5dadd05ad22004f2e402df--