Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B75E7D562 for ; Tue, 21 Aug 2012 01:59:16 +0000 (UTC) Received: (qmail 39134 invoked by uid 500); 21 Aug 2012 01:59:16 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 39088 invoked by uid 500); 21 Aug 2012 01:59:16 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 39079 invoked by uid 99); 21 Aug 2012 01:59:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Aug 2012 01:59:16 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [24.173.238.83] (HELO mail.potomacfusion.com) (24.173.238.83) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Aug 2012 01:59:10 +0000 Received: from PFI-MAIL.PFI.LOCAL ([192.168.100.7]) by pfi-mail ([192.168.100.7]) with mapi; Mon, 20 Aug 2012 20:58:48 -0500 From: David Garcia To: "user@giraph.apache.org" Date: Mon, 20 Aug 2012 20:56:51 -0500 Subject: RE: cannot run Giraph trunk with Hadoop 2.0.0-alpha Thread-Topic: cannot run Giraph trunk with Hadoop 2.0.0-alpha Thread-Index: Ac1/MUU8d/Fs+9QsQVSWkVjoQlKyMgADvX0y Message-ID: <688D8B269DCBDE44A466DC43D403624C06AAD21429@pfi-mail> References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 You can remove this error by recursively removing _bsp folder from the zook= eeper file system...and then running the job again. Probably should remove= folder from hdfs too. ________________________________________ From: Johnny Zhang [xiaoyuz@cloudera.com] Sent: Monday, August 20, 2012 6:59 PM To: user@giraph.apache.org Subject: Re: cannot run Giraph trunk with Hadoop 2.0.0-alpha sorry for wide distribution, I further check the folder '_bsp/_defaultZkMa= nagerDir/job_1344903945125_0032' exists, and it has one sub folder '_bsp/_= defaultZkManagerDir/job_1344903945125_0032/_task' and another file inside, = so the hdfs file permission should not be a issue. but not sure why Giraph = still complain '_bsp/_defaultZkManagerDir/job_1344903945125_0032/_zkServer = does not exist'. Does Zookeeper needs further configuration? Or any other possible reason ca= nnot create _zkServer folder ? Thanks, Johnny On Mon, Aug 20, 2012 at 11:59 AM, Johnny Zhang > wrote: Alessandro: Thanks for reminding me on that. Now I can run the pagerank example success= fully, though I still get one zookeeper server related exception. Here is p= art of the log: 12/08/20 11:56:44 WARN mapreduce.Job: Error reading task output Server retu= rned HTTP response code: 400 for URL: http://cs-10-20-76-76.cloud.cloudera.= com:8080/tasklog?plaintext=3Dtrue&attemptid=3Dattempt_1344903945125_0032_m_= 000002_2&filter=3Dstdout 12/08/20 11:56:44 WARN mapreduce.Job: Error reading task output Server retu= rned HTTP response code: 400 for URL: http://cs-10-20-76-76.cloud.cloudera.= com:8080/tasklog?plaintext=3Dtrue&attemptid=3Dattempt_1344903945125_0032_m_= 000002_2&filter=3Dstderr 12/08/20 11:56:44 INFO mapreduce.Job: Task Id : attempt_1344903945125_0032_= m_000001_2, Status : FAILED Error: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp= /_defaultZkManagerDir/job_1344903945125_0032/_zkServer does not exist. at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperMa= nager.java:749) at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:320) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:570) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio= n.java:1232) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/jo= b_1344903945125_0032/_zkServer does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileS= ystem.java:365) at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperMa= nager.java:708) ... 9 more 12/08/20 11:56:44 WARN mapreduce.Job: Error reading task output Server retu= rned HTTP response code: 400 for URL: http://cs-10-20-76-76.cloud.cloudera.= com:8080/tasklog?plaintext=3Dtrue&attemptid=3Dattempt_1344903945125_0032_m_= 000001_2&filter=3Dstdout 12/08/20 11:56:44 WARN mapreduce.Job: Error reading task output Server retu= rned HTTP response code: 400 for URL: http://cs-10-20-76-76.cloud.cloudera.= com:8080/tasklog?plaintext=3Dtrue&attemptid=3Dattempt_1344903945125_0032_m_= 000001_2&filter=3Dstderr 12/08/20 11:56:45 INFO mapreduce.Job: Job job_1344903945125_0032 failed wit= h state FAILED due to: 12/08/20 11:56:45 INFO mapreduce.Job: Counters: 28 File System Counters FILE: Number of bytes read=3D120 FILE: Number of bytes written=3D49450 FILE: Number of read operations=3D0 FILE: Number of large read operations=3D0 FILE: Number of write operations=3D0 HDFS: Number of bytes read=3D44 HDFS: Number of bytes written=3D0 HDFS: Number of read operations=3D8 HDFS: Number of large read operations=3D0 HDFS: Number of write operations=3D3 Job Counters Failed map tasks=3D10 Launched map tasks=3D13 Other local map tasks=3D13 Total time spent by all maps in occupied slots (ms)=3D692328 Total time spent by all reduces in occupied slots (ms)=3D0 Map-Reduce Framework Map input records=3D0 Map output records=3D0 Input split bytes=3D44 Spilled Records=3D0 Failed Shuffles=3D0 Merged Map outputs=3D0 GC time elapsed (ms)=3D34 CPU time spent (ms)=3D450 Physical memory (bytes) snapshot=3D96169984 Virtual memory (bytes) snapshot=3D1599012864 Total committed heap usage (bytes)=3D76087296 File Input Format Counters Bytes Read=3D0 File Output Format Counters Bytes Written=3D0 Thanks, Johnny On Mon, Aug 20, 2012 at 11:47 AM, Alessandro Presta > wrote: Looks like you compiled for hadoop 0.20.203, which had a different API (tha= t's why we have to use Munge). Can you try recompiling with the hadoop_2.0.= 0 profile? From: Johnny Zhang > Reply-To: "user@giraph.apache.org" > Date: Monday, August 20, 2012 7:31 PM To: "user@giraph.apache.org" > Subject: cannot run Giraph trunk with Hadoop 2.0.0-alpha Hi, all: I am trying to run Giraph trunk with Hadoop 2.0.0-alpha. I am getting below error when I run a page rank example job with 3 workers. # hadoop jar target/giraph-0.2-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-depe= ndencies.jar org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V = 50000000 -w 3 12/08/20 11:10:38 WARN mapred.JobConf: The variable mapred.child.ulimit is = no longer used. 12/08/20 11:10:38 INFO benchmark.PageRankBenchmark: Using class org.apache.= giraph.benchmark.PageRankBenchmark 12/08/20 11:10:38 WARN conf.Configuration: mapred.job.tracker is deprecated= . Instead, use mapreduce.jobtracker.address 12/08/20 11:10:38 WARN conf.Configuration: mapred.job.map.memory.mb is depr= ecated. Instead, use mapreduce.map.memory.mb 12/08/20 11:10:38 WARN conf.Configuration: mapred.job.reduce.memory.mb is d= eprecated. Instead, use mapreduce.reduce.memory.mb 12/08/20 11:10:38 WARN conf.Configuration: mapred.map.tasks.speculative.exe= cution is deprecated. Instead, use mapreduce.map.speculative Exception in thread "main" java.lang.IncompatibleClassChangeError: Found in= terface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.j= ava:43) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:41= 1) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.= java:326) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio= n.java:1232) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244) at org.apache.giraph.graph.GiraphJob.run(GiraphJob.java:714) at org.apache.giraph.benchmark.PageRankBenchmark.run(PageRankBenchmark.java= :150) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.benchmark.PageRankBenchmark.main(PageRankBenchmark.jav= a:164) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav= a:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor= Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) My $HADOOP_MAPRED_HOME and $JAVA_HOME is set up correctly, could anyone tel= l me if I need to setup anything else? Thanks a lot. Johnny