hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward <edw...@udanax.org>
Subject Re: Error with fastgen input
Date Thu, 28 Feb 2013 08:27:59 GMT
yes i'll check again

Sent from my iPhone

On Feb 28, 2013, at 5:18 PM, Thomas Jungblut <thomas.jungblut@gmail.com> wrote:

> Can you verify an observation for me please?
> 
> 2 files are created from fastgen, part-00000 and part-00001, both ~2.2kb
> sized.
> In the below partition directory, there is only a single 5.56kb file.
> 
> Is it intended for the partitioner to write a single file if you configured
> two?
> It even reads it as a two files, strange huh?
> 
> 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com>
> 
>> Will have a look into it.
>> 
>> gen fastgen 100 10 /tmp/randomgraph 1
>> pagerank /tmp/randomgraph /tmp/pageout
>> 
>> did work for me the last time I profiled, maybe the partitioning doesn't
>> partition correctly with the input or something else.
>> 
>> 
>> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>> 
>> Fastgen input seems not work for graph examples.
>>> 
>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar
>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen fastgen 100 10
>>> /tmp/randomgraph 2
>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop library for your platform... using builtin-java classes
>>> where applicable
>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient: Running job: job_localrunner_0001
>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner: Setting up a new barrier for 2
>>> tasks!
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Current supersteps number: 0
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The total number of supersteps: 0
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: Counters: 3
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> org.apache.hama.bsp.JobInProgress$JobCounter
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:     SUPERSTEPS=0
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=2
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:     TASK_OUTPUT_RECORDS=100
>>> Job Finished in 3.212 seconds
>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar
>>> examples/target/hama-examples-0.7.0-SNAPSHOT
>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar
>>> hama-examples-0.7.0-SNAPSHOT.jar
>>> edward@edward-virtualBox:~/workspace/hama-trunk$ bin/hama jar
>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar pagerank
>>> /tmp/randomgraph /tmp/pageour
>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop library for your platform... using builtin-java classes
>>> where applicable
>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths to process
>>> : 2
>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat: Total input paths to process
>>> : 2
>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient: Running job: job_localrunner_0001
>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner: Setting up a new barrier for 2
>>> tasks!
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Current supersteps number: 1
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The total number of supersteps: 1
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Counters: 6
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> org.apache.hama.bsp.JobInProgress$JobCounter
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     SUPERSTEPS=1
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=2
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=4
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     IO_BYTES_READ=4332
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=14
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=100
>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat: Total input paths to process
>>> : 2
>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: Running job: job_localrunner_0001
>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner: Setting up a new barrier for 2
>>> tasks!
>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are loaded into
>>> local:1
>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50 vertices are loaded into
>>> local:0
>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner: Exception during BSP
>>> execution!
>>> java.lang.IllegalArgumentException: Messages must never be behind the
>>> vertex in ID! Current Message ID: 1 vs. 50
>>>        at
>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279)
>>>        at
>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225)
>>>        at
>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129)
>>>        at
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
>>>        at
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>>>        at
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>>>        at
>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>        at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>        at
>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>        at java.lang.Thread.run(Thread.java:722)
>>> 
>>> 
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>> 
>> 

Mime
View raw message