hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Error with fastgen input
Date Wed, 13 Mar 2013 14:09:35 GMT
>>> in loadVertices? Maybe consider feature for coupling storage in user space
>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way
>>> partitioned or non-partitioned by partitioner, can keep vertices sorted
>>> with a single read and single write on every peer.

And, as I commented JIRA ticket, I think we can't use messaging system
for sorting vertices within partition files.

On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> P.S., (number of splits = number of partitions) is really confuse to
> me. Even though blocks number is equal to desired tasks number, data
> should be re-partitioned again.
>
> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> Indeed. If there are already partitioned input files (unsorted) and so
>> user want to skip pre-partitioning phase, it should be handled in
>> GraphJobRunner BSP program. Actually, I still don't know why
>> re-partitioned files need to be Sorted. It's only about
>> GraphJobRunner.
>>
>>> partitioning. (This is outside the scope of graphs. We can have a dedicated
>>> partitioning superstep for graph applications).
>>
>> Sorry. I don't understand exactly yet. Do you mean just a partitioning
>> job based on superstep API?
>>
>> By default, 100 tasks will be assigned for partitioning job.
>> Partitioning job will create 1,000 partitions. Thus, we can execute
>> the Graph job with 1,000 tasks.
>>
>> Let's assume that a input sequence file is 20GB (100 blocks). If I
>> want to run with 1,000 tasks, what happens?
>>
>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <surajsmenon@apache.org> wrote:
>>> I am responding on this thread because of better continuity for
>>> conversation. We cannot expect the partitions to be sorted every time. When
>>> the number of splits = number of partitions and partitioning is switched
>>> off by user[HAMA-561], the partitions would not be sorted. Can we do this
>>> in loadVertices? Maybe consider feature for coupling storage in user space
>>> with BSP Messaging[HAMA-734] can avoid double reads and writes. This way
>>> partitioned or non-partitioned by partitioner, can keep vertices sorted
>>> with a single read and single write on every peer.
>>>
>>> Just clearing confusion if any regarding superstep injection for
>>> partitioning. (This is outside the scope of graphs. We can have a dedicated
>>> partitioning superstep for graph applications).
>>> Say there are x splits and y number of tasks configured by user.
>>>
>>> if x > y
>>> The y tasks are scheduled with x of them having each of the x splits and
>>> the remaining with no resource local to them. Then the partitioning
>>> superstep redistributes the partitions among them to create local
>>> partitions. Now the question is can we re-initialize a peer's input based
>>> on this new local part of partition?
>>>
>>> if y > x
>>> works as it works today.
>>>
>>> Just putting my points in brainstorming.
>>>
>>> -Suraj
>>>
>>>
>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>>>
>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744
>>>>
>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <edwardyoon@apache.org>
>>>> wrote:
>>>> > Additionally,
>>>> >
>>>> >> spilling queue and sorted spilling queue, can we inject the partitioning
>>>> >> superstep as the first superstep and use local memory?
>>>> >
>>>> > Can we execute different number of tasks per superstep?
>>>> >
>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <edwardyoon@apache.org>
>>>> wrote:
>>>> >>> For graph processing, the partitioned files that result from the
>>>> >>> partitioning job must be sorted. Currently only the partition files in
>>>> >>
>>>> >> I see.
>>>> >>
>>>> >>> For other partitionings and with regard to our superstep API, Suraj's
>>>> idea
>>>> >>> of injecting a preprocessing superstep that partitions the stuff into
>>>> our
>>>> >>> messaging system is actually the best.
>>>> >>
>>>> >> BTW, if some garbage objects can be accumulated in partitioning step,
>>>> >> separated partitioning job may not be bad idea. Is there some special
>>>> >> reason?
>>>> >>
>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut
>>>> >> <thomas.jungblut@gmail.com> wrote:
>>>> >>> For graph processing, the partitioned files that result from the
>>>> >>> partitioning job must be sorted. Currently only the partition files in
>>>> >>> itself are sorted, thus more tasks result in not sorted data in the
>>>> >>> completed file. This only applies for the graph processing package.
>>>> >>> So as Suraj told, it would be much more simpler to solve this via
>>>> >>> messaging, once it is scalable (it will be very very scalable!). So the
>>>> >>> GraphJobRunner can be partitioning the stuff with a single superstep in
>>>> >>> setup() as it was before ages ago. The messaging must be sorted anyway
>>>> for
>>>> >>> the algorithm so this is a nice side effect and saves us the
>>>> partitioning
>>>> >>> job for graph processing.
>>>> >>>
>>>> >>> For other partitionings and with regard to our superstep API, Suraj's
>>>> idea
>>>> >>> of injecting a preprocessing superstep that partitions the stuff into
>>>> our
>>>> >>> messaging system is actually the best.
>>>> >>>
>>>> >>>
>>>> >>> 2013/3/6 Suraj Menon <surajsmenon@apache.org>
>>>> >>>
>>>> >>>> No, the partitions we write locally need not be sorted. Sorry for the
>>>> >>>> confusion. The Superstep injection is possible with Superstep API.
>>>> There
>>>> >>>> are few enhancements needed to make it simpler after I last worked on
>>>> it.
>>>> >>>> We can then look into partitioning superstep being executed before the
>>>> >>>> setup of first superstep of submitted job. I think it is feasible.
>>>> >>>>
>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <edwardyoon@apache.org
>>>> >>>> >wrote:
>>>> >>>>
>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the
>>>> >>>> partitioning
>>>> >>>> > > superstep as the first superstep and use local memory?
>>>> >>>> >
>>>> >>>> > Actually, I wanted to add something before calling BSP.setup()
>>>> method
>>>> >>>> > to avoid execute additional BSP job. But, in my opinion, current is
>>>> >>>> > enough. I think, we need to collect more experiences of input
>>>> >>>> > partitioning on large environments. I'll do.
>>>> >>>> >
>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like?
>>>> >>>> >
>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon <
>>>> surajsmenon@apache.org>
>>>> >>>> > wrote:
>>>> >>>> > > Sorry, I am increasing the scope here to outside graph module.
>>>> When we
>>>> >>>> > have
>>>> >>>> > > spilling queue and sorted spilling queue, can we inject the
>>>> >>>> partitioning
>>>> >>>> > > superstep as the first superstep and use local memory?
>>>> >>>> > > Today we have partitioning job within a job and are creating two
>>>> copies
>>>> >>>> > of
>>>> >>>> > > data on HDFS. This could be really costly. Is it possible to
>>>> create or
>>>> >>>> > > redistribute the partitions on local memory and initialize the
>>>> record
>>>> >>>> > > reader there?
>>>> >>>> > > The user can run a separate job give in examples area to
>>>> explicitly
>>>> >>>> > > repartition the data on HDFS. The deployment question is how much
>>>> of
>>>> >>>> disk
>>>> >>>> > > space gets allocated for local memory usage? Would it be a safe
>>>> >>>> approach
>>>> >>>> > > with the limitations?
>>>> >>>> > >
>>>> >>>> > > -Suraj
>>>> >>>> > >
>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut
>>>> >>>> > > <thomas.jungblut@gmail.com>wrote:
>>>> >>>> > >
>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add this to
>>>> the
>>>> >>>> > >> partitioner pretty easily.
>>>> >>>> > >>
>>>> >>>> > >> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>>> >>>> > >>
>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary to be
>>>> Sorted?
>>>> >>>> > >> >
>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut
>>>> >>>> > >> > <thomas.jungblut@gmail.com> wrote:
>>>> >>>> > >> > > Now I get how the partitioning works, obviously if you merge
>>>> n
>>>> >>>> > sorted
>>>> >>>> > >> > files
>>>> >>>> > >> > > by just appending to each other, this will result in totally
>>>> >>>> > unsorted
>>>> >>>> > >> > data
>>>> >>>> > >> > > ;-)
>>>> >>>> > >> > > Why didn't you solve this via messaging?
>>>> >>>> > >> > >
>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com>
>>>> >>>> > >> > >
>>>> >>>> > >> > >> Seems that they are not correctly sorted:
>>>> >>>> > >> > >>
>>>> >>>> > >> > >> vertexID: 50
>>>> >>>> > >> > >> vertexID: 52
>>>> >>>> > >> > >> vertexID: 54
>>>> >>>> > >> > >> vertexID: 56
>>>> >>>> > >> > >> vertexID: 58
>>>> >>>> > >> > >> vertexID: 61
>>>> >>>> > >> > >> ...
>>>> >>>> > >> > >> vertexID: 78
>>>> >>>> > >> > >> vertexID: 81
>>>> >>>> > >> > >> vertexID: 83
>>>> >>>> > >> > >> vertexID: 85
>>>> >>>> > >> > >> ...
>>>> >>>> > >> > >> vertexID: 94
>>>> >>>> > >> > >> vertexID: 96
>>>> >>>> > >> > >> vertexID: 98
>>>> >>>> > >> > >> vertexID: 1
>>>> >>>> > >> > >> vertexID: 10
>>>> >>>> > >> > >> vertexID: 12
>>>> >>>> > >> > >> vertexID: 14
>>>> >>>> > >> > >> vertexID: 16
>>>> >>>> > >> > >> vertexID: 18
>>>> >>>> > >> > >> vertexID: 21
>>>> >>>> > >> > >> vertexID: 23
>>>> >>>> > >> > >> vertexID: 25
>>>> >>>> > >> > >> vertexID: 27
>>>> >>>> > >> > >> vertexID: 29
>>>> >>>> > >> > >> vertexID: 3
>>>> >>>> > >> > >>
>>>> >>>> > >> > >> So this won't work then correctly...
>>>> >>>> > >> > >>
>>>> >>>> > >> > >>
>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com>
>>>> >>>> > >> > >>
>>>> >>>> > >> > >>> sure, have fun on your holidays.
>>>> >>>> > >> > >>>
>>>> >>>> > >> > >>>
>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>>> >>>> > >> > >>>
>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do. March 1 is
>>>> >>>> > holiday[1]
>>>> >>>> > >> so
>>>> >>>> > >> > >>>> I'll appear next week.
>>>> >>>> > >> > >>>>
>>>> >>>> > >> > >>>> 1.
>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea
>>>> >>>> > >> > >>>>
>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut
>>>> >>>> > >> > >>>> <thomas.jungblut@gmail.com> wrote:
>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't observe if all
>>>> >>>> items
>>>> >>>> > >> were
>>>> >>>> > >> > >>>> added.
>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic of the ID
>>>> into
>>>> >>>> > the
>>>> >>>> > >> > >>>> fastgen,
>>>> >>>> > >> > >>>> > want to have a look into it?
>>>> >>>> > >> > >>>> >
>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>>> >>>> > >> > >>>> >
>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate adjacency
>>>> >>>> matrix
>>>> >>>> > >> into
>>>> >>>> > >> > >>>> >> multiple files.
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas Jungblut
>>>> >>>> > >> > >>>> >> <thomas.jungblut@gmail.com> wrote:
>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned correctly?
>>>> >>>> > >> > >>>> >> >
>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>>> >>>> > >> > >>>> >> >
>>>> >>>> > >> > >>>> >> >> It looks like a bug.
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al
>>>> >>>> > >> /tmp/randomgraph/
>>>> >>>> > >> > >>>> >> >> total 44
>>>> >>>> > >> > >>>> >> >> drwxrwxr-x  3 edward edward  4096  2월 28 18:03 .
>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root   root   20480  2월 28 18:04 ..
>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx  1 edward edward  2243  2월 28 18:01
>>>> part-00000
>>>> >>>> > >> > >>>> >> >> -rw-rw-r--  1 edward edward    28  2월 28 18:01
>>>> >>>> > .part-00000.crc
>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx  1 edward edward  2251  2월 28 18:01
>>>> part-00001
>>>> >>>> > >> > >>>> >> >> -rw-rw-r--  1 edward edward    28  2월 28 18:01
>>>> >>>> > .part-00001.crc
>>>> >>>> > >> > >>>> >> >> drwxrwxr-x  2 edward edward  4096  2월 28 18:03
>>>> partitions
>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls -al
>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/
>>>> >>>> > >> > >>>> >> >> total 24
>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096  2월 28 18:03 .
>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096  2월 28 18:03 ..
>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932  2월 28 18:03
>>>> part-00000
>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward   32  2월 28 18:03
>>>> >>>> > .part-00000.crc
>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955  2월 28 18:03
>>>> part-00001
>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward   32  2월 28 18:03
>>>> >>>> > .part-00001.crc
>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward <
>>>> >>>> edward@udanax.org
>>>> >>>> > >
>>>> >>>> > >> > wrote:
>>>> >>>> > >> > >>>> >> >> > yes i'll check again
>>>> >>>> > >> > >>>> >> >> >
>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone
>>>> >>>> > >> > >>>> >> >> >
>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas Jungblut <
>>>> >>>> > >> > >>>> >> thomas.jungblut@gmail.com>
>>>> >>>> > >> > >>>> >> >> wrote:
>>>> >>>> > >> > >>>> >> >> >
>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me please?
>>>> >>>> > >> > >>>> >> >> >>
>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen, part-00000 and
>>>> >>>> > >> part-00001,
>>>> >>>> > >> > >>>> both
>>>> >>>> > >> > >>>> >> ~2.2kb
>>>> >>>> > >> > >>>> >> >> >> sized.
>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there is only a
>>>> >>>> single
>>>> >>>> > >> > 5.56kb
>>>> >>>> > >> > >>>> file.
>>>> >>>> > >> > >>>> >> >> >>
>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to write a
>>>> single
>>>> >>>> > file
>>>> >>>> > >> if
>>>> >>>> > >> > you
>>>> >>>> > >> > >>>> >> >> configured
>>>> >>>> > >> > >>>> >> >> >> two?
>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange huh?
>>>> >>>> > >> > >>>> >> >> >>
>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut <
>>>> thomas.jungblut@gmail.com>
>>>> >>>> > >> > >>>> >> >> >>
>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it.
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1
>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I profiled, maybe
>>>> the
>>>> >>>> > >> > >>>> partitioning
>>>> >>>> > >> > >>>> >> >> doesn't
>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or something
>>>> else.
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org
>>>> >
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph examples.
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>>>> :~/workspace/hama-trunk$
>>>> >>>> > >> bin/hama
>>>> >>>> > >> > jar
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen
>>>> >>>> > >> > fastgen
>>>> >>>> > >> > >>>> 100 10
>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN util.NativeCodeLoader:
>>>> Unable
>>>> >>>> > to
>>>> >>>> > >> > load
>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform...
>>>> using
>>>> >>>> > >> > builtin-java
>>>> >>>> > >> > >>>> >> classes
>>>> >>>> > >> > >>>> >> >> >>>> where applicable
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.BSPJobClient:
>>>> Running
>>>> >>>> job:
>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO bsp.LocalBSPRunner:
>>>> Setting
>>>> >>>> up
>>>> >>>> > a
>>>> >>>> > >> new
>>>> >>>> > >> > >>>> barrier
>>>> >>>> > >> > >>>> >> >> for 2
>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> Current
>>>> >>>> > >> supersteps
>>>> >>>> > >> > >>>> >> number: 0
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient: The
>>>> total
>>>> >>>> > number
>>>> >>>> > >> > of
>>>> >>>> > >> > >>>> >> >> supersteps: 0
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> Counters: 3
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> >>>> > SUPERSTEPS=0
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> >>>> > >> > LAUNCHED_TASKS=2
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100
>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds
>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>>>> :~/workspace/hama-trunk$
>>>> >>>> > >> bin/hama
>>>> >>>> > >> > jar
>>>> >>>> > >> > >>>> >> >> >>>> examples/target/hama-examples-0.7.0-SNAPSHOT
>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar
>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar
>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>>>> :~/workspace/hama-trunk$
>>>> >>>> > >> bin/hama
>>>> >>>> > >> > jar
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar
>>>> >>>> > pagerank
>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN util.NativeCodeLoader:
>>>> Unable
>>>> >>>> > to
>>>> >>>> > >> > load
>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your platform...
>>>> using
>>>> >>>> > >> > builtin-java
>>>> >>>> > >> > >>>> >> classes
>>>> >>>> > >> > >>>> >> >> >>>> where applicable
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat:
>>>> Total
>>>> >>>> > input
>>>> >>>> > >> > paths
>>>> >>>> > >> > >>>> to
>>>> >>>> > >> > >>>> >> >> process
>>>> >>>> > >> > >>>> >> >> >>>> : 2
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO bsp.FileInputFormat:
>>>> Total
>>>> >>>> > input
>>>> >>>> > >> > paths
>>>> >>>> > >> > >>>> to
>>>> >>>> > >> > >>>> >> >> process
>>>> >>>> > >> > >>>> >> >> >>>> : 2
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.BSPJobClient:
>>>> Running
>>>> >>>> job:
>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO bsp.LocalBSPRunner:
>>>> Setting
>>>> >>>> up
>>>> >>>> > a
>>>> >>>> > >> new
>>>> >>>> > >> > >>>> barrier
>>>> >>>> > >> > >>>> >> >> for 2
>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> Current
>>>> >>>> > >> supersteps
>>>> >>>> > >> > >>>> >> number: 1
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient: The
>>>> total
>>>> >>>> > number
>>>> >>>> > >> > of
>>>> >>>> > >> > >>>> >> >> supersteps: 1
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> Counters: 6
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.JobInProgress$JobCounter
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > SUPERSTEPS=1
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > LAUNCHED_TASKS=2
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> >> >> >>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > SUPERSTEP_SUM=4
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> IO_BYTES_READ=4332
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.FileInputFormat:
>>>> Total
>>>> >>>> > input
>>>> >>>> > >> > paths
>>>> >>>> > >> > >>>> to
>>>> >>>> > >> > >>>> >> >> process
>>>> >>>> > >> > >>>> >> >> >>>> : 2
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.BSPJobClient:
>>>> Running
>>>> >>>> job:
>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO bsp.LocalBSPRunner:
>>>> Setting
>>>> >>>> up
>>>> >>>> > a
>>>> >>>> > >> new
>>>> >>>> > >> > >>>> barrier
>>>> >>>> > >> > >>>> >> >> for 2
>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50
>>>> >>>> > vertices
>>>> >>>> > >> > are
>>>> >>>> > >> > >>>> loaded
>>>> >>>> > >> > >>>> >> >> into
>>>> >>>> > >> > >>>> >> >> >>>> local:1
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO graph.GraphJobRunner: 50
>>>> >>>> > vertices
>>>> >>>> > >> > are
>>>> >>>> > >> > >>>> loaded
>>>> >>>> > >> > >>>> >> >> into
>>>> >>>> > >> > >>>> >> >> >>>> local:0
>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR bsp.LocalBSPRunner:
>>>> >>>> Exception
>>>> >>>> > >> > during
>>>> >>>> > >> > >>>> BSP
>>>> >>>> > >> > >>>> >> >> >>>> execution!
>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException: Messages
>>>> must
>>>> >>>> > never
>>>> >>>> > >> be
>>>> >>>> > >> > >>>> behind
>>>> >>>> > >> > >>>> >> the
>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1 vs. 50
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> >
>>>> >>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> >
>>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>>
>>>> >>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> >
>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> > >>
>>>> >>>> >
>>>> >>>>
>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> > >>
>>>> >>>> >
>>>> >>>>
>>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>>
>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> >
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>>
>>>> >>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> > >>
>>>> >>>> >
>>>> >>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> >>>> > >> > >>>> >> >> >>>>        at
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> >
>>>> >>>> > >>
>>>> >>>> >
>>>> >>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> >>>> > >> > >>>> >> >> >>>>        at java.lang.Thread.run(Thread.java:722)
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >> >>>>
>>>> >>>> > >> > >>>> >> >> >>>> --
>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon
>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >> >>>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >> >> --
>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon
>>>> >>>> > >> > >>>> >> >> @eddieyoon
>>>> >>>> > >> > >>>> >> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>> >> --
>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon
>>>> >>>> > >> > >>>> >> @eddieyoon
>>>> >>>> > >> > >>>> >>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> > >>>>
>>>> >>>> > >> > >>>> --
>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon
>>>> >>>> > >> > >>>> @eddieyoon
>>>> >>>> > >> > >>>>
>>>> >>>> > >> > >>>
>>>> >>>> > >> > >>>
>>>> >>>> > >> > >>
>>>> >>>> > >> >
>>>> >>>> > >> >
>>>> >>>> > >> >
>>>> >>>> > >> > --
>>>> >>>> > >> > Best Regards, Edward J. Yoon
>>>> >>>> > >> > @eddieyoon
>>>> >>>> > >> >
>>>> >>>> > >>
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > --
>>>> >>>> > Best Regards, Edward J. Yoon
>>>> >>>> > @eddieyoon
>>>> >>>> >
>>>> >>>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards, Edward J. Yoon
>>>> >> @eddieyoon
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Best Regards, Edward J. Yoon
>>>> > @eddieyoon
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> @eddieyoon
>>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message