hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Error with fastgen input
Date Thu, 28 Feb 2013 09:34:55 GMT
I guess, it's a bug of fastgen, when generate adjacency matrix into
multiple files.

On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut
<thomas.jungblut@gmail.com> wrote:
> You have two files, are they partitioned correctly?
>
> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>
>> It looks like a bug.
>>
>> edward@udanax:~/workspace/hama-trunk$ ls -al /tmp/randomgraph/
>> total 44
>> drwxrwxr-x  3 edward edward  4096  2월 28 18:03 .
>> drwxrwxrwt 19 root   root   20480  2월 28 18:04 ..
>> -rwxrwxrwx  1 edward edward  2243  2월 28 18:01 part-00000
>> -rw-rw-r--  1 edward edward    28  2월 28 18:01 .part-00000.crc
>> -rwxrwxrwx  1 edward edward  2251  2월 28 18:01 part-00001
>> -rw-rw-r--  1 edward edward    28  2월 28 18:01 .part-00001.crc
>> drwxrwxr-x  2 edward edward  4096  2월 28 18:03 partitions
>> edward@udanax:~/workspace/hama-trunk$ ls -al /tmp/randomgraph/partitions/
>> total 24
>> drwxrwxr-x 2 edward edward 4096  2월 28 18:03 .
>> drwxrwxr-x 3 edward edward 4096  2월 28 18:03 ..
>> -rwxrwxrwx 1 edward edward 2932  2월 28 18:03 part-00000
>> -rw-rw-r-- 1 edward edward   32  2월 28 18:03 .part-00000.crc
>> -rwxrwxrwx 1 edward edward 2955  2월 28 18:03 part-00001
>> -rw-rw-r-- 1 edward edward   32  2월 28 18:03 .part-00001.crc
>> edward@udanax:~/workspace/hama-trunk$
>>
>>
>> On Thu, Feb 28, 2013 at 5:27 PM, Edward <edward@udanax.org> wrote:
>> > yes i'll check again
>> >
>> > Sent from my iPhone
>> >
>> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut <thomas.jungblut@gmail.com>
>> wrote:
>> >
>> >> Can you verify an observation for me please?
>> >>
>> >> 2 files are created from fastgen, part-00000 and part-00001, both ~2.2kb
>> >> sized.
>> >> In the below partition directory, there is only a single 5.56kb file.
>> >>
>> >> Is it intended for the partitioner to write a single file if you
>> configured
>> >> two?
>> >> It even reads it as a two files, strange huh?
>> >>
>> >> 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com>
>> >>
>> >>> Will have a look into it.
>> >>>
>> >>> gen fastgen 100 10 /tmp/randomgraph 1
>> >>> pagerank /tmp/randomgraph /tmp/pageout
>> >>>
>> >>> did work for me the last time I profiled, maybe the partitioning
>> doesn't
>> >>> partition correctly with the input or something else.
>> >>>
>> >>>
>> >>> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>> >>>
>> >>> Fastgen input seems not work for graph examples.
>> >>>>
>> >>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar
>> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen fastgen 100
10
>> >>>> /tmp/randomgraph 2
>> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: Unable to load
>> >>>> native-hadoop library for your platform... using builtin-java classes
>> >>>> where applicable
>> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: Running job:
>> job_localrunner_0001
>> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: Setting up a new barrier
>> for 2
>> >>>> tasks!
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Current supersteps number:
0
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The total number of
>> supersteps: 0
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Counters: 3
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>> >>>> org.apache.hama.bsp.JobInProgress$JobCounter
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:     SUPERSTEPS=0
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=2
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:     TASK_OUTPUT_RECORDS=100
>> >>>> Job Finished in 3.212 seconds
>> >>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar
>> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT
>> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar
>> >>>> hama-examples-0.7.0-SNAPSHOT.jar
>> >>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar
>> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar pagerank
>> >>>> /tmp/randomgraph /tmp/pageour
>> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: Unable to load
>> >>>> native-hadoop library for your platform... using builtin-java classes
>> >>>> where applicable
>> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths to
>> process
>> >>>> : 2
>> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths to
>> process
>> >>>> : 2
>> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: Running job:
>> job_localrunner_0001
>> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: Setting up a new barrier
>> for 2
>> >>>> tasks!
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Current supersteps number:
1
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The total number of
>> supersteps: 1
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Counters: 6
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>> >>>> org.apache.hama.bsp.JobInProgress$JobCounter
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     SUPERSTEPS=1
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=2
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=4
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     IO_BYTES_READ=4332
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=14
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=100
>> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: Total input paths to
>> process
>> >>>> : 2
>> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Running job:
>> job_localrunner_0001
>> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: Setting up a new barrier
>> for 2
>> >>>> tasks!
>> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are loaded
>> into
>> >>>> local:1
>> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are loaded
>> into
>> >>>> local:0
>> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: Exception during BSP
>> >>>> execution!
>> >>>> java.lang.IllegalArgumentException: Messages must never be behind
the
>> >>>> vertex in ID! Current Message ID: 1 vs. 50
>> >>>>        at
>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279)
>> >>>>        at
>> >>>>
>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225)
>> >>>>        at
>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129)
>> >>>>        at
>> >>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
>> >>>>        at
>> >>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>> >>>>        at
>> >>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>> >>>>        at
>> >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>>>        at
>> >>>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>>>        at
>> >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >>>>        at
>> >>>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >>>>        at
>> >>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >>>>        at java.lang.Thread.run(Thread.java:722)
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Best Regards, Edward J. Yoon
>> >>>> @eddieyoon
>> >>>
>> >>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message