hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Error with fastgen input
Date Wed, 13 Mar 2013 14:00:53 GMT
P.S., (number of splits = number of partitions) is really confuse to
me. Even though blocks number is equal to desired tasks number, data
should be re-partitioned again.

On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Indeed. If there are already partitioned input files (unsorted) and so
> user want to skip pre-partitioning phase, it should be handled in
> GraphJobRunner BSP program. Actually, I still don't know why
> re-partitioned files need to be Sorted. It's only about
> GraphJobRunner.
>
>> partitioning. (This is outside the scope of graphs. We can have a dedicated
>> partitioning superstep for graph applications).
>
> Sorry. I don't understand exactly yet. Do you mean just a partitioning
> job based on superstep API?
>
> By default, 100 tasks will be assigned for partitioning job.
> Partitioning job will create 1,000 partitions. Thus, we can execute
> the Graph job with 1,000 tasks.
>
> Let's assume that a input sequence file is 20GB (100 blocks). If I
> want to run with 1,000 tasks, what happens?
>
> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <surajsmenon@apache.org> wrote:
>> I am responding on this thread because of better continuity for
>> conversation. We cannot expect the partitions to be sorted every time. When
>> the number of splits = number of partitions and partitioning is switched
>> off by user[HAMA-561], the partitions would not be sorted. Can we do this
>> in loadVertices? Maybe consider feature for coupling storage in user space
>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way
>> partitioned or non-partitioned by partitioner, can keep vertices sorted
>> with a single read and single write on every peer.
>>
>> Just clearing confusion if any regarding superstep injection for
>> partitioning. (This is outside the scope of graphs. We can have a dedicated
>> partitioning superstep for graph applications).
>> Say there are x splits and y number of tasks configured by user.
>>
>> if x > y
>> The y tasks are scheduled with x of them having each of the x splits and
>> the remaining with no resource local to them. Then the partitioning
>> superstep redistributes the partitions among them to create local
>> partitions. Now the question is can we re-initialize a peer's input based
>> on this new local part of partition?
>>
>> if y > x
>> works as it works today.
>>
>> Just putting my points in brainstorming.
>>
>> -Suraj
>>
>>
>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>>
>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744
>>>
>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <edwardyoon@apache.org>
>>> wrote:
>>> > Additionally,
>>> >
>>> >> spilling queue and sorted spilling queue, can we inject the partitioning
>>> >> superstep as the first superstep and use local memory?
>>> >
>>> > Can we execute different number of tasks per superstep?
>>> >
>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <edwardyoon@apache.org>
>>> wrote:
>>> >>> For graph processing, the partitioned files that result from the
>>> >>> partitioning job must be sorted. Currently only the partition files
in
>>> >>
>>> >> I see.
>>> >>
>>> >>> For other partitionings and with regard to our superstep API, Suraj's
>>> idea
>>> >>> of injecting a preprocessing superstep that partitions the stuff
into
>>> our
>>> >>> messaging system is actually the best.
>>> >>
>>> >> BTW, if some garbage objects can be accumulated in partitioning step,
>>> >> separated partitioning job may not be bad idea. Is there some special
>>> >> reason?
>>> >>
>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut
>>> >> <thomas.jungblut@gmail.com> wrote:
>>> >>> For graph processing, the partitioned files that result from the
>>> >>> partitioning job must be sorted. Currently only the partition files
in
>>> >>> itself are sorted, thus more tasks result in not sorted data in
the
>>> >>> completed file. This only applies for the graph processing package.
>>> >>> So as Suraj told, it would be much more simpler to solve this via
>>> >>> messaging, once it is scalable (it will be very very scalable!).
So the
>>> >>> GraphJobRunner can be partitioning the stuff with a single superstep
in
>>> >>> setup() as it was before ages ago. The messaging must be sorted
anyway
>>> for
>>> >>> the algorithm so this is a nice side effect and saves us the
>>> partitioning
>>> >>> job for graph processing.
>>> >>>
>>> >>> For other partitionings and with regard to our superstep API, Suraj's
>>> idea
>>> >>> of injecting a preprocessing superstep that partitions the stuff
into
>>> our
>>> >>> messaging system is actually the best.
>>> >>>
>>> >>>
>>> >>> 2013/3/6 Suraj Menon <surajsmenon@apache.org>
>>> >>>
>>> >>>> No, the partitions we write locally need not be sorted. Sorry
for the
>>> >>>> confusion. The Superstep injection is possible with Superstep
API.
>>> There
>>> >>>> are few enhancements needed to make it simpler after I last
worked on
>>> it.
>>> >>>> We can then look into partitioning superstep being executed
before the
>>> >>>> setup of first superstep of submitted job. I think it is feasible.
>>> >>>>
>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <edwardyoon@apache.org
>>> >>>> >wrote:
>>> >>>>
>>> >>>> > > spilling queue and sorted spilling queue, can we inject
the
>>> >>>> partitioning
>>> >>>> > > superstep as the first superstep and use local memory?
>>> >>>> >
>>> >>>> > Actually, I wanted to add something before calling BSP.setup()
>>> method
>>> >>>> > to avoid execute additional BSP job. But, in my opinion,
current is
>>> >>>> > enough. I think, we need to collect more experiences of
input
>>> >>>> > partitioning on large environments. I'll do.
>>> >>>> >
>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like?
>>> >>>> >
>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon <
>>> surajsmenon@apache.org>
>>> >>>> > wrote:
>>> >>>> > > Sorry, I am increasing the scope here to outside graph
module.
>>> When we
>>> >>>> > have
>>> >>>> > > spilling queue and sorted spilling queue, can we inject
the
>>> >>>> partitioning
>>> >>>> > > superstep as the first superstep and use local memory?
>>> >>>> > > Today we have partitioning job within a job and are
creating two
>>> copies
>>> >>>> > of
>>> >>>> > > data on HDFS. This could be really costly. Is it possible
to
>>> create or
>>> >>>> > > redistribute the partitions on local memory and initialize
the
>>> record
>>> >>>> > > reader there?
>>> >>>> > > The user can run a separate job give in examples area
to
>>> explicitly
>>> >>>> > > repartition the data on HDFS. The deployment question
is how much
>>> of
>>> >>>> disk
>>> >>>> > > space gets allocated for local memory usage? Would
it be a safe
>>> >>>> approach
>>> >>>> > > with the limitations?
>>> >>>> > >
>>> >>>> > > -Suraj
>>> >>>> > >
>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut
>>> >>>> > > <thomas.jungblut@gmail.com>wrote:
>>> >>>> > >
>>> >>>> > >> yes. Once Suraj added merging of sorted files
we can add this to
>>> the
>>> >>>> > >> partitioner pretty easily.
>>> >>>> > >>
>>> >>>> > >> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>> >>>> > >>
>>> >>>> > >> > Eh,..... btw, is re-partitioned data really
necessary to be
>>> Sorted?
>>> >>>> > >> >
>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut
>>> >>>> > >> > <thomas.jungblut@gmail.com> wrote:
>>> >>>> > >> > > Now I get how the partitioning works,
obviously if you merge
>>> n
>>> >>>> > sorted
>>> >>>> > >> > files
>>> >>>> > >> > > by just appending to each other, this
will result in totally
>>> >>>> > unsorted
>>> >>>> > >> > data
>>> >>>> > >> > > ;-)
>>> >>>> > >> > > Why didn't you solve this via messaging?
>>> >>>> > >> > >
>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com>
>>> >>>> > >> > >
>>> >>>> > >> > >> Seems that they are not correctly
sorted:
>>> >>>> > >> > >>
>>> >>>> > >> > >> vertexID: 50
>>> >>>> > >> > >> vertexID: 52
>>> >>>> > >> > >> vertexID: 54
>>> >>>> > >> > >> vertexID: 56
>>> >>>> > >> > >> vertexID: 58
>>> >>>> > >> > >> vertexID: 61
>>> >>>> > >> > >> ...
>>> >>>> > >> > >> vertexID: 78
>>> >>>> > >> > >> vertexID: 81
>>> >>>> > >> > >> vertexID: 83
>>> >>>> > >> > >> vertexID: 85
>>> >>>> > >> > >> ...
>>> >>>> > >> > >> vertexID: 94
>>> >>>> > >> > >> vertexID: 96
>>> >>>> > >> > >> vertexID: 98
>>> >>>> > >> > >> vertexID: 1
>>> >>>> > >> > >> vertexID: 10
>>> >>>> > >> > >> vertexID: 12
>>> >>>> > >> > >> vertexID: 14
>>> >>>> > >> > >> vertexID: 16
>>> >>>> > >> > >> vertexID: 18
>>> >>>> > >> > >> vertexID: 21
>>> >>>> > >> > >> vertexID: 23
>>> >>>> > >> > >> vertexID: 25
>>> >>>> > >> > >> vertexID: 27
>>> >>>> > >> > >> vertexID: 29
>>> >>>> > >> > >> vertexID: 3
>>> >>>> > >> > >>
>>> >>>> > >> > >> So this won't work then correctly...
>>> >>>> > >> > >>
>>> >>>> > >> > >>
>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com>
>>> >>>> > >> > >>
>>> >>>> > >> > >>> sure, have fun on your holidays.
>>> >>>> > >> > >>>
>>> >>>> > >> > >>>
>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>> >>>> > >> > >>>
>>> >>>> > >> > >>>> Sure, but if you can fix
quickly, please do. March 1 is
>>> >>>> > holiday[1]
>>> >>>> > >> so
>>> >>>> > >> > >>>> I'll appear next week.
>>> >>>> > >> > >>>>
>>> >>>> > >> > >>>> 1.
>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea
>>> >>>> > >> > >>>>
>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at
6:36 PM, Thomas Jungblut
>>> >>>> > >> > >>>> <thomas.jungblut@gmail.com>
wrote:
>>> >>>> > >> > >>>> > Maybe 50 is missing
from the file, didn't observe if all
>>> >>>> items
>>> >>>> > >> were
>>> >>>> > >> > >>>> added.
>>> >>>> > >> > >>>> > As far as I remember,
I copy/pasted the logic of the ID
>>> into
>>> >>>> > the
>>> >>>> > >> > >>>> fastgen,
>>> >>>> > >> > >>>> > want to have a look
into it?
>>> >>>> > >> > >>>> >
>>> >>>> > >> > >>>> > 2013/2/28 Edward J.
Yoon <edwardyoon@apache.org>
>>> >>>> > >> > >>>> >
>>> >>>> > >> > >>>> >> I guess, it's a
bug of fastgen, when generate adjacency
>>> >>>> matrix
>>> >>>> > >> into
>>> >>>> > >> > >>>> >> multiple files.
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>> >> On Thu, Feb 28,
2013 at 6:29 PM, Thomas Jungblut
>>> >>>> > >> > >>>> >> <thomas.jungblut@gmail.com>
wrote:
>>> >>>> > >> > >>>> >> > You have two
files, are they partitioned correctly?
>>> >>>> > >> > >>>> >> >
>>> >>>> > >> > >>>> >> > 2013/2/28
Edward J. Yoon <edwardyoon@apache.org>
>>> >>>> > >> > >>>> >> >
>>> >>>> > >> > >>>> >> >> It looks
like a bug.
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$
ls -al
>>> >>>> > >> /tmp/randomgraph/
>>> >>>> > >> > >>>> >> >> total
44
>>> >>>> > >> > >>>> >> >> drwxrwxr-x
 3 edward edward  4096  2월 28 18:03 .
>>> >>>> > >> > >>>> >> >> drwxrwxrwt
19 root   root   20480  2월 28 18:04 ..
>>> >>>> > >> > >>>> >> >> -rwxrwxrwx
 1 edward edward  2243  2월 28 18:01
>>> part-00000
>>> >>>> > >> > >>>> >> >> -rw-rw-r--
 1 edward edward    28  2월 28 18:01
>>> >>>> > .part-00000.crc
>>> >>>> > >> > >>>> >> >> -rwxrwxrwx
 1 edward edward  2251  2월 28 18:01
>>> part-00001
>>> >>>> > >> > >>>> >> >> -rw-rw-r--
 1 edward edward    28  2월 28 18:01
>>> >>>> > .part-00001.crc
>>> >>>> > >> > >>>> >> >> drwxrwxr-x
 2 edward edward  4096  2월 28 18:03
>>> partitions
>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$
ls -al
>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/
>>> >>>> > >> > >>>> >> >> total
24
>>> >>>> > >> > >>>> >> >> drwxrwxr-x
2 edward edward 4096  2월 28 18:03 .
>>> >>>> > >> > >>>> >> >> drwxrwxr-x
3 edward edward 4096  2월 28 18:03 ..
>>> >>>> > >> > >>>> >> >> -rwxrwxrwx
1 edward edward 2932  2월 28 18:03
>>> part-00000
>>> >>>> > >> > >>>> >> >> -rw-rw-r--
1 edward edward   32  2월 28 18:03
>>> >>>> > .part-00000.crc
>>> >>>> > >> > >>>> >> >> -rwxrwxrwx
1 edward edward 2955  2월 28 18:03
>>> part-00001
>>> >>>> > >> > >>>> >> >> -rw-rw-r--
1 edward edward   32  2월 28 18:03
>>> >>>> > .part-00001.crc
>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >> >> On Thu,
Feb 28, 2013 at 5:27 PM, Edward <
>>> >>>> edward@udanax.org
>>> >>>> > >
>>> >>>> > >> > wrote:
>>> >>>> > >> > >>>> >> >> > yes
i'll check again
>>> >>>> > >> > >>>> >> >> >
>>> >>>> > >> > >>>> >> >> > Sent
from my iPhone
>>> >>>> > >> > >>>> >> >> >
>>> >>>> > >> > >>>> >> >> > On
Feb 28, 2013, at 5:18 PM, Thomas Jungblut <
>>> >>>> > >> > >>>> >> thomas.jungblut@gmail.com>
>>> >>>> > >> > >>>> >> >> wrote:
>>> >>>> > >> > >>>> >> >> >
>>> >>>> > >> > >>>> >> >> >>
Can you verify an observation for me please?
>>> >>>> > >> > >>>> >> >> >>
>>> >>>> > >> > >>>> >> >> >>
2 files are created from fastgen, part-00000 and
>>> >>>> > >> part-00001,
>>> >>>> > >> > >>>> both
>>> >>>> > >> > >>>> >> ~2.2kb
>>> >>>> > >> > >>>> >> >> >>
sized.
>>> >>>> > >> > >>>> >> >> >>
In the below partition directory, there is only a
>>> >>>> single
>>> >>>> > >> > 5.56kb
>>> >>>> > >> > >>>> file.
>>> >>>> > >> > >>>> >> >> >>
>>> >>>> > >> > >>>> >> >> >>
Is it intended for the partitioner to write a
>>> single
>>> >>>> > file
>>> >>>> > >> if
>>> >>>> > >> > you
>>> >>>> > >> > >>>> >> >> configured
>>> >>>> > >> > >>>> >> >> >>
two?
>>> >>>> > >> > >>>> >> >> >>
It even reads it as a two files, strange huh?
>>> >>>> > >> > >>>> >> >> >>
>>> >>>> > >> > >>>> >> >> >>
2013/2/28 Thomas Jungblut <
>>> thomas.jungblut@gmail.com>
>>> >>>> > >> > >>>> >> >> >>
>>> >>>> > >> > >>>> >> >> >>>
Will have a look into it.
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >> >>>
gen fastgen 100 10 /tmp/randomgraph 1
>>> >>>> > >> > >>>> >> >> >>>
pagerank /tmp/randomgraph /tmp/pageout
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >> >>>
did work for me the last time I profiled, maybe
>>> the
>>> >>>> > >> > >>>> partitioning
>>> >>>> > >> > >>>> >> >> doesn't
>>> >>>> > >> > >>>> >> >> >>>
partition correctly with the input or something
>>> else.
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >> >>>
2013/2/28 Edward J. Yoon <edwardyoon@apache.org
>>> >
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >> >>>
Fastgen input seems not work for graph examples.
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >> >>>>
edward@edward-virtualBox
>>> :~/workspace/hama-trunk$
>>> >>>> > >> bin/hama
>>> >>>> > >> > jar
>>> >>>> > >> > >>>> >> >> >>>>
>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen
>>> >>>> > >> > fastgen
>>> >>>> > >> > >>>> 100 10
>>> >>>> > >> > >>>> >> >> >>>>
/tmp/randomgraph 2
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:02 WARN util.NativeCodeLoader:
>>> Unable
>>> >>>> > to
>>> >>>> > >> > load
>>> >>>> > >> > >>>> >> >> >>>>
native-hadoop library for your platform...
>>> using
>>> >>>> > >> > builtin-java
>>> >>>> > >> > >>>> >> classes
>>> >>>> > >> > >>>> >> >> >>>>
where applicable
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:03 INFO bsp.BSPJobClient:
>>> Running
>>> >>>> job:
>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:03 INFO bsp.LocalBSPRunner:
>>> Setting
>>> >>>> up
>>> >>>> > a
>>> >>>> > >> new
>>> >>>> > >> > >>>> barrier
>>> >>>> > >> > >>>> >> >> for 2
>>> >>>> > >> > >>>> >> >> >>>>
tasks!
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> Current
>>> >>>> > >> supersteps
>>> >>>> > >> > >>>> >> number: 0
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient: The
>>> total
>>> >>>> > number
>>> >>>> > >> > of
>>> >>>> > >> > >>>> >> >> supersteps:
0
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> Counters: 3
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> >> >> >>>>
org.apache.hama.bsp.JobInProgress$JobCounter
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> >>>> > SUPERSTEPS=0
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> >>>> > >> > LAUNCHED_TASKS=2
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> >> >> >>>>
org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100
>>> >>>> > >> > >>>> >> >> >>>>
Job Finished in 3.212 seconds
>>> >>>> > >> > >>>> >> >> >>>>
edward@edward-virtualBox
>>> :~/workspace/hama-trunk$
>>> >>>> > >> bin/hama
>>> >>>> > >> > jar
>>> >>>> > >> > >>>> >> >> >>>>
examples/target/hama-examples-0.7.0-SNAPSHOT
>>> >>>> > >> > >>>> >> >> >>>>
hama-examples-0.7.0-SNAPSHOT-javadoc.jar
>>> >>>> > >> > >>>> >> >> >>>>
hama-examples-0.7.0-SNAPSHOT.jar
>>> >>>> > >> > >>>> >> >> >>>>
edward@edward-virtualBox
>>> :~/workspace/hama-trunk$
>>> >>>> > >> bin/hama
>>> >>>> > >> > jar
>>> >>>> > >> > >>>> >> >> >>>>
>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar
>>> >>>> > pagerank
>>> >>>> > >> > >>>> >> >> >>>>
/tmp/randomgraph /tmp/pageour
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:29 WARN util.NativeCodeLoader:
>>> Unable
>>> >>>> > to
>>> >>>> > >> > load
>>> >>>> > >> > >>>> >> >> >>>>
native-hadoop library for your platform...
>>> using
>>> >>>> > >> > builtin-java
>>> >>>> > >> > >>>> >> classes
>>> >>>> > >> > >>>> >> >> >>>>
where applicable
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:29 INFO bsp.FileInputFormat:
>>> Total
>>> >>>> > input
>>> >>>> > >> > paths
>>> >>>> > >> > >>>> to
>>> >>>> > >> > >>>> >> >> process
>>> >>>> > >> > >>>> >> >> >>>>
: 2
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:29 INFO bsp.FileInputFormat:
>>> Total
>>> >>>> > input
>>> >>>> > >> > paths
>>> >>>> > >> > >>>> to
>>> >>>> > >> > >>>> >> >> process
>>> >>>> > >> > >>>> >> >> >>>>
: 2
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:30 INFO bsp.BSPJobClient:
>>> Running
>>> >>>> job:
>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:30 INFO bsp.LocalBSPRunner:
>>> Setting
>>> >>>> up
>>> >>>> > a
>>> >>>> > >> new
>>> >>>> > >> > >>>> barrier
>>> >>>> > >> > >>>> >> >> for 2
>>> >>>> > >> > >>>> >> >> >>>>
tasks!
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> Current
>>> >>>> > >> supersteps
>>> >>>> > >> > >>>> >> number: 1
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient: The
>>> total
>>> >>>> > number
>>> >>>> > >> > of
>>> >>>> > >> > >>>> >> >> supersteps:
1
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> Counters: 6
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> >> >> >>>>
org.apache.hama.bsp.JobInProgress$JobCounter
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > SUPERSTEPS=1
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > LAUNCHED_TASKS=2
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> >> >> >>>>
org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > SUPERSTEP_SUM=4
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> IO_BYTES_READ=4332
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.FileInputFormat:
>>> Total
>>> >>>> > input
>>> >>>> > >> > paths
>>> >>>> > >> > >>>> to
>>> >>>> > >> > >>>> >> >> process
>>> >>>> > >> > >>>> >> >> >>>>
: 2
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>> Running
>>> >>>> job:
>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO bsp.LocalBSPRunner:
>>> Setting
>>> >>>> up
>>> >>>> > a
>>> >>>> > >> new
>>> >>>> > >> > >>>> barrier
>>> >>>> > >> > >>>> >> >> for 2
>>> >>>> > >> > >>>> >> >> >>>>
tasks!
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO graph.GraphJobRunner: 50
>>> >>>> > vertices
>>> >>>> > >> > are
>>> >>>> > >> > >>>> loaded
>>> >>>> > >> > >>>> >> >> into
>>> >>>> > >> > >>>> >> >> >>>>
local:1
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 INFO graph.GraphJobRunner: 50
>>> >>>> > vertices
>>> >>>> > >> > are
>>> >>>> > >> > >>>> loaded
>>> >>>> > >> > >>>> >> >> into
>>> >>>> > >> > >>>> >> >> >>>>
local:0
>>> >>>> > >> > >>>> >> >> >>>>
13/02/28 10:32:33 ERROR bsp.LocalBSPRunner:
>>> >>>> Exception
>>> >>>> > >> > during
>>> >>>> > >> > >>>> BSP
>>> >>>> > >> > >>>> >> >> >>>>
execution!
>>> >>>> > >> > >>>> >> >> >>>>
java.lang.IllegalArgumentException: Messages
>>> must
>>> >>>> > never
>>> >>>> > >> be
>>> >>>> > >> > >>>> behind
>>> >>>> > >> > >>>> >> the
>>> >>>> > >> > >>>> >> >> >>>>
vertex in ID! Current Message ID: 1 vs. 50
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> >
>>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> >
>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>>
>>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> >
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> > >>
>>> >>>> >
>>> >>>>
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> > >>
>>> >>>> >
>>> >>>>
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>>
>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> >
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>>
>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> > >>
>>> >>>> >
>>> >>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> >>>> > >> > >>>> >> >> >>>>
       at
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> >
>>> >>>> > >>
>>> >>>> >
>>> >>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> >>>> > >> > >>>> >> >> >>>>
       at java.lang.Thread.run(Thread.java:722)
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>> > >> > >>>> >> >> >>>>
--
>>> >>>> > >> > >>>> >> >> >>>>
Best Regards, Edward J. Yoon
>>> >>>> > >> > >>>> >> >> >>>>
@eddieyoon
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >> >>>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >> >> --
>>> >>>> > >> > >>>> >> >> Best Regards,
Edward J. Yoon
>>> >>>> > >> > >>>> >> >> @eddieyoon
>>> >>>> > >> > >>>> >> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>> >> --
>>> >>>> > >> > >>>> >> Best Regards, Edward
J. Yoon
>>> >>>> > >> > >>>> >> @eddieyoon
>>> >>>> > >> > >>>> >>
>>> >>>> > >> > >>>>
>>> >>>> > >> > >>>>
>>> >>>> > >> > >>>>
>>> >>>> > >> > >>>> --
>>> >>>> > >> > >>>> Best Regards, Edward J.
Yoon
>>> >>>> > >> > >>>> @eddieyoon
>>> >>>> > >> > >>>>
>>> >>>> > >> > >>>
>>> >>>> > >> > >>>
>>> >>>> > >> > >>
>>> >>>> > >> >
>>> >>>> > >> >
>>> >>>> > >> >
>>> >>>> > >> > --
>>> >>>> > >> > Best Regards, Edward J. Yoon
>>> >>>> > >> > @eddieyoon
>>> >>>> > >> >
>>> >>>> > >>
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > --
>>> >>>> > Best Regards, Edward J. Yoon
>>> >>>> > @eddieyoon
>>> >>>> >
>>> >>>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards, Edward J. Yoon
>>> >> @eddieyoon
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > @eddieyoon
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message