Return-Path: X-Original-To: apmail-hama-dev-archive@www.apache.org Delivered-To: apmail-hama-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 02E0BE609 for ; Thu, 28 Feb 2013 10:38:34 +0000 (UTC) Received: (qmail 7355 invoked by uid 500); 28 Feb 2013 10:38:33 -0000 Delivered-To: apmail-hama-dev-archive@hama.apache.org Received: (qmail 7333 invoked by uid 500); 28 Feb 2013 10:38:33 -0000 Mailing-List: contact dev-help@hama.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hama.apache.org Delivered-To: mailing list dev@hama.apache.org Received: (qmail 7316 invoked by uid 99); 28 Feb 2013 10:38:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Feb 2013 10:38:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of thomas.jungblut@gmail.com designates 209.85.216.51 as permitted sender) Received: from [209.85.216.51] (HELO mail-qa0-f51.google.com) (209.85.216.51) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Feb 2013 10:38:28 +0000 Received: by mail-qa0-f51.google.com with SMTP id cr7so1227500qab.17 for ; Thu, 28 Feb 2013 02:38:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=ikOl2w4Vk3czu+/y5P2GVx6WE3nPL3J8VI1rnWmP2T4=; b=Kyqt8dRE8xVbdeUWJj1XNmoEqnwoysp/hMBbzqP1WhBcQfNPLLUbuT/Q4ywLEsfdgT vOVT6raJDTFlFDd/Qy3H0ePFOYmc73kQYaj4YMyhz6pFflTBvI/0ZyRWWOkI78j1Y9Ko U7LCGHgQ/5M+SS99R+Y29lJDz9h9ZHP7LFK1yM/RMv1f3uqYOzwE9v4AudPyKnVe14+R rTxuKAtAxZmDHauou/3SA18MWUIBNiHqZWvspHAMf57hUzES7EGMNWPaqwOi8CW3T/iw PQCeq0bkGJHsT+3uK003IprVqsHcXhF7VHqfAlZEN3nolrd0snEeMOahGYDxbQUTx+aT WGFA== MIME-Version: 1.0 X-Received: by 10.229.114.209 with SMTP id f17mr1968014qcq.156.1362047888017; Thu, 28 Feb 2013 02:38:08 -0800 (PST) Received: by 10.49.63.136 with HTTP; Thu, 28 Feb 2013 02:38:07 -0800 (PST) In-Reply-To: References: Date: Thu, 28 Feb 2013 11:38:07 +0100 Message-ID: Subject: Re: Error with fastgen input From: Thomas Jungblut To: dev@hama.apache.org Content-Type: multipart/alternative; boundary=00235429d6c488720504d6c67cdc X-Virus-Checked: Checked by ClamAV on apache.org --00235429d6c488720504d6c67cdc Content-Type: text/plain; charset=EUC-KR Content-Transfer-Encoding: quoted-printable Seems that they are not correctly sorted: vertexID: 50 vertexID: 52 vertexID: 54 vertexID: 56 vertexID: 58 vertexID: 61 ... vertexID: 78 vertexID: 81 vertexID: 83 vertexID: 85 ... vertexID: 94 vertexID: 96 vertexID: 98 vertexID: 1 vertexID: 10 vertexID: 12 vertexID: 14 vertexID: 16 vertexID: 18 vertexID: 21 vertexID: 23 vertexID: 25 vertexID: 27 vertexID: 29 vertexID: 3 So this won't work then correctly... 2013/2/28 Thomas Jungblut > sure, have fun on your holidays. > > > 2013/2/28 Edward J. Yoon > >> Sure, but if you can fix quickly, please do. March 1 is holiday[1] so >> I'll appear next week. >> >> 1. http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >> >> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut >> wrote: >> > Maybe 50 is missing from the file, didn't observe if all items were >> added. >> > As far as I remember, I copy/pasted the logic of the ID into the >> fastgen, >> > want to have a look into it? >> > >> > 2013/2/28 Edward J. Yoon >> > >> >> I guess, it's a bug of fastgen, when generate adjacency matrix into >> >> multiple files. >> >> >> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut >> >> wrote: >> >> > You have two files, are they partitioned correctly? >> >> > >> >> > 2013/2/28 Edward J. Yoon >> >> > >> >> >> It looks like a bug. >> >> >> >> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al /tmp/randomgraph/ >> >> >> total 44 >> >> >> drwxrwxr-x 3 edward edward 4096 2=BF=F9 28 18:03 . >> >> >> drwxrwxrwt 19 root root 20480 2=BF=F9 28 18:04 .. >> >> >> -rwxrwxrwx 1 edward edward 2243 2=BF=F9 28 18:01 part-00000 >> >> >> -rw-rw-r-- 1 edward edward 28 2=BF=F9 28 18:01 .part-00000.cr= c >> >> >> -rwxrwxrwx 1 edward edward 2251 2=BF=F9 28 18:01 part-00001 >> >> >> -rw-rw-r-- 1 edward edward 28 2=BF=F9 28 18:01 .part-00001.cr= c >> >> >> drwxrwxr-x 2 edward edward 4096 2=BF=F9 28 18:03 partitions >> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al >> >> /tmp/randomgraph/partitions/ >> >> >> total 24 >> >> >> drwxrwxr-x 2 edward edward 4096 2=BF=F9 28 18:03 . >> >> >> drwxrwxr-x 3 edward edward 4096 2=BF=F9 28 18:03 .. >> >> >> -rwxrwxrwx 1 edward edward 2932 2=BF=F9 28 18:03 part-00000 >> >> >> -rw-rw-r-- 1 edward edward 32 2=BF=F9 28 18:03 .part-00000.crc >> >> >> -rwxrwxrwx 1 edward edward 2955 2=BF=F9 28 18:03 part-00001 >> >> >> -rw-rw-r-- 1 edward edward 32 2=BF=F9 28 18:03 .part-00001.crc >> >> >> edward@udanax:~/workspace/hama-trunk$ >> >> >> >> >> >> >> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward wrote: >> >> >> > yes i'll check again >> >> >> > >> >> >> > Sent from my iPhone >> >> >> > >> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut < >> >> thomas.jungblut@gmail.com> >> >> >> wrote: >> >> >> > >> >> >> >> Can you verify an observation for me please? >> >> >> >> >> >> >> >> 2 files are created from fastgen, part-00000 and part-00001, bo= th >> >> ~2.2kb >> >> >> >> sized. >> >> >> >> In the below partition directory, there is only a single 5.56kb >> file. >> >> >> >> >> >> >> >> Is it intended for the partitioner to write a single file if yo= u >> >> >> configured >> >> >> >> two? >> >> >> >> It even reads it as a two files, strange huh? >> >> >> >> >> >> >> >> 2013/2/28 Thomas Jungblut >> >> >> >> >> >> >> >>> Will have a look into it. >> >> >> >>> >> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1 >> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout >> >> >> >>> >> >> >> >>> did work for me the last time I profiled, maybe the partitioni= ng >> >> >> doesn't >> >> >> >>> partition correctly with the input or something else. >> >> >> >>> >> >> >> >>> >> >> >> >>> 2013/2/28 Edward J. Yoon >> >> >> >>> >> >> >> >>> Fastgen input seems not work for graph examples. >> >> >> >>>> >> >> >> >>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar >> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen fastgen >> 100 10 >> >> >> >>>> /tmp/randomgraph 2 >> >> >> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: Unable to load >> >> >> >>>> native-hadoop library for your platform... using builtin-java >> >> classes >> >> >> >>>> where applicable >> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: Running job: >> >> >> job_localrunner_0001 >> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: Setting up a new >> barrier >> >> >> for 2 >> >> >> >>>> tasks! >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Current supersteps >> >> number: 0 >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The total number of >> >> >> supersteps: 0 >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Counters: 3 >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: SUPERSTEPS=3D0 >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: LAUNCHED_TASKS= =3D2 >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: >> >> TASK_OUTPUT_RECORDS=3D100 >> >> >> >>>> Job Finished in 3.212 seconds >> >> >> >>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar >> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT >> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar >> >> >> >>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar >> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar pagerank >> >> >> >>>> /tmp/randomgraph /tmp/pageour >> >> >> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: Unable to load >> >> >> >>>> native-hadoop library for your platform... using builtin-java >> >> classes >> >> >> >>>> where applicable >> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths >> to >> >> >> process >> >> >> >>>> : 2 >> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths >> to >> >> >> process >> >> >> >>>> : 2 >> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: Running job: >> >> >> job_localrunner_0001 >> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: Setting up a new >> barrier >> >> >> for 2 >> >> >> >>>> tasks! >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Current supersteps >> >> number: 1 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The total number of >> >> >> supersteps: 1 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Counters: 6 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: SUPERSTEPS=3D1 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: LAUNCHED_TASKS= =3D2 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: SUPERSTEP_SUM=3D= 4 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: IO_BYTES_READ=3D= 4332 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS= =3D14 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: >> TASK_INPUT_RECORDS=3D100 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: Total input paths >> to >> >> >> process >> >> >> >>>> : 2 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Running job: >> >> >> job_localrunner_0001 >> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: Setting up a new >> barrier >> >> >> for 2 >> >> >> >>>> tasks! >> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are >> loaded >> >> >> into >> >> >> >>>> local:1 >> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are >> loaded >> >> >> into >> >> >> >>>> local:0 >> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: Exception during >> BSP >> >> >> >>>> execution! >> >> >> >>>> java.lang.IllegalArgumentException: Messages must never be >> behind >> >> the >> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50 >> >> >> >>>> at >> >> >> >>>> >> >> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> >> >> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225= ) >> >> >> >>>> at >> >> >> >>>> >> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256= ) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:28= 6) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:21= 1) >> >> >> >>>> at >> >> >> >>>> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> >> >>>> at >> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> >> >>>> at >> >> >> >>>> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> >> >>>> at >> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1110) >> >> >> >>>> at >> >> >> >>>> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:603) >> >> >> >>>> at java.lang.Thread.run(Thread.java:722) >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> -- >> >> >> >>>> Best Regards, Edward J. Yoon >> >> >> >>>> @eddieyoon >> >> >> >>> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Best Regards, Edward J. Yoon >> >> >> @eddieyoon >> >> >> >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> @eddieyoon >> >> >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> > > --00235429d6c488720504d6c67cdc--