hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Error with fastgen input
Date Thu, 14 Mar 2013 08:04:54 GMT
P.S., Please don't say like that.

No decisions made yet. And if someone have a question or missed
something, you have to try to explain here. Because this is a open
source. Anyone can't say "don't touch trunk bc I'm working on it".

On Thu, Mar 14, 2013 at 4:37 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Sorry for my quick and dirty style small patches.
>
> However, we should work together in parallel. Please share here if
> there are some progresses.
>
> On Thu, Mar 14, 2013 at 3:46 PM, Thomas Jungblut
> <thomas.jungblut@gmail.com> wrote:
>> Hi Edward,
>>
>> before you run riot on all along the codebase, Suraj ist currently working
>> on that stuff- don't make it more difficult for him rebasing all his
>> patches the whole time.
>> He has the plan so that we made to make the stuff working, his part is
>> currently missing. So don't try to muddle arround there, it will make this
>> take longer than already needed.
>>
>>
>>
>> 2013/3/14 Edward J. Yoon <edwardyoon@apache.org>
>>
>>> Personally, I would like to solve this issue by touching
>>> DiskVerticesInfo. If we write sorted sub-sets of vertices into
>>> multiple files, we can avoid huge memory consumption.
>>>
>>> If we want to sort partitioned data using messaging system, idea
>>> should be collected.
>>>
>>> On Thu, Mar 14, 2013 at 10:31 AM, Edward J. Yoon <edwardyoon@apache.org>
>>> wrote:
>>> > Oh, now I get how iterate() works. HAMA-704 is nicely written.
>>> >
>>> > On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon <edwardyoon@apache.org>
>>> wrote:
>>> >> I'm reading changes of HAMA-704 again. As a result of adding
>>> >> DiskVerticesInfo, vertices list is needed to be sorted. I'm not sure
>>> >> but I think this approach will bring more disadvantages than
>>> >> advantages.
>>> >>
>>> >> On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon <edwardyoon@apache.org>
>>> wrote:
>>> >>>>>> in loadVertices? Maybe consider feature for coupling storage in
>>> user space
>>> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes.
>>> This way
>>> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices
>>> sorted
>>> >>>>>> with a single read and single write on every peer.
>>> >>>
>>> >>> And, as I commented JIRA ticket, I think we can't use messaging system
>>> >>> for sorting vertices within partition files.
>>> >>>
>>> >>> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon <
>>> edwardyoon@apache.org> wrote:
>>> >>>> P.S., (number of splits = number of partitions) is really confuse to
>>> >>>> me. Even though blocks number is equal to desired tasks number, data
>>> >>>> should be re-partitioned again.
>>> >>>>
>>> >>>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon <
>>> edwardyoon@apache.org> wrote:
>>> >>>>> Indeed. If there are already partitioned input files (unsorted) and
>>> so
>>> >>>>> user want to skip pre-partitioning phase, it should be handled in
>>> >>>>> GraphJobRunner BSP program. Actually, I still don't know why
>>> >>>>> re-partitioned files need to be Sorted. It's only about
>>> >>>>> GraphJobRunner.
>>> >>>>>
>>> >>>>>> partitioning. (This is outside the scope of graphs. We can have a
>>> dedicated
>>> >>>>>> partitioning superstep for graph applications).
>>> >>>>>
>>> >>>>> Sorry. I don't understand exactly yet. Do you mean just a
>>> partitioning
>>> >>>>> job based on superstep API?
>>> >>>>>
>>> >>>>> By default, 100 tasks will be assigned for partitioning job.
>>> >>>>> Partitioning job will create 1,000 partitions. Thus, we can execute
>>> >>>>> the Graph job with 1,000 tasks.
>>> >>>>>
>>> >>>>> Let's assume that a input sequence file is 20GB (100 blocks). If I
>>> >>>>> want to run with 1,000 tasks, what happens?
>>> >>>>>
>>> >>>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon <surajsmenon@apache.org>
>>> wrote:
>>> >>>>>> I am responding on this thread because of better continuity for
>>> >>>>>> conversation. We cannot expect the partitions to be sorted every
>>> time. When
>>> >>>>>> the number of splits = number of partitions and partitioning is
>>> switched
>>> >>>>>> off by user[HAMA-561], the partitions would not be sorted. Can we
>>> do this
>>> >>>>>> in loadVertices? Maybe consider feature for coupling storage in
>>> user space
>>> >>>>>> with BSP Messaging[HAMA-734] can avoid double reads and writes.
>>> This way
>>> >>>>>> partitioned or non-partitioned by partitioner, can keep vertices
>>> sorted
>>> >>>>>> with a single read and single write on every peer.
>>> >>>>>>
>>> >>>>>> Just clearing confusion if any regarding superstep injection for
>>> >>>>>> partitioning. (This is outside the scope of graphs. We can have a
>>> dedicated
>>> >>>>>> partitioning superstep for graph applications).
>>> >>>>>> Say there are x splits and y number of tasks configured by user.
>>> >>>>>>
>>> >>>>>> if x > y
>>> >>>>>> The y tasks are scheduled with x of them having each of the x
>>> splits and
>>> >>>>>> the remaining with no resource local to them. Then the partitioning
>>> >>>>>> superstep redistributes the partitions among them to create local
>>> >>>>>> partitions. Now the question is can we re-initialize a peer's input
>>> based
>>> >>>>>> on this new local part of partition?
>>> >>>>>>
>>> >>>>>> if y > x
>>> >>>>>> works as it works today.
>>> >>>>>>
>>> >>>>>> Just putting my points in brainstorming.
>>> >>>>>>
>>> >>>>>> -Suraj
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon <
>>> edwardyoon@apache.org>wrote:
>>> >>>>>>
>>> >>>>>>> I just filed here https://issues.apache.org/jira/browse/HAMA-744
>>> >>>>>>>
>>> >>>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon <
>>> edwardyoon@apache.org>
>>> >>>>>>> wrote:
>>> >>>>>>> > Additionally,
>>> >>>>>>> >
>>> >>>>>>> >> spilling queue and sorted spilling queue, can we inject the
>>> partitioning
>>> >>>>>>> >> superstep as the first superstep and use local memory?
>>> >>>>>>> >
>>> >>>>>>> > Can we execute different number of tasks per superstep?
>>> >>>>>>> >
>>> >>>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoon <
>>> edwardyoon@apache.org>
>>> >>>>>>> wrote:
>>> >>>>>>> >>> For graph processing, the partitioned files that result from
>>> the
>>> >>>>>>> >>> partitioning job must be sorted. Currently only the partition
>>> files in
>>> >>>>>>> >>
>>> >>>>>>> >> I see.
>>> >>>>>>> >>
>>> >>>>>>> >>> For other partitionings and with regard to our superstep API,
>>> Suraj's
>>> >>>>>>> idea
>>> >>>>>>> >>> of injecting a preprocessing superstep that partitions the
>>> stuff into
>>> >>>>>>> our
>>> >>>>>>> >>> messaging system is actually the best.
>>> >>>>>>> >>
>>> >>>>>>> >> BTW, if some garbage objects can be accumulated in partitioning
>>> step,
>>> >>>>>>> >> separated partitioning job may not be bad idea. Is there some
>>> special
>>> >>>>>>> >> reason?
>>> >>>>>>> >>
>>> >>>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungblut
>>> >>>>>>> >> <thomas.jungblut@gmail.com> wrote:
>>> >>>>>>> >>> For graph processing, the partitioned files that result from
>>> the
>>> >>>>>>> >>> partitioning job must be sorted. Currently only the partition
>>> files in
>>> >>>>>>> >>> itself are sorted, thus more tasks result in not sorted data
>>> in the
>>> >>>>>>> >>> completed file. This only applies for the graph processing
>>> package.
>>> >>>>>>> >>> So as Suraj told, it would be much more simpler to solve this
>>> via
>>> >>>>>>> >>> messaging, once it is scalable (it will be very very
>>> scalable!). So the
>>> >>>>>>> >>> GraphJobRunner can be partitioning the stuff with a single
>>> superstep in
>>> >>>>>>> >>> setup() as it was before ages ago. The messaging must be
>>> sorted anyway
>>> >>>>>>> for
>>> >>>>>>> >>> the algorithm so this is a nice side effect and saves us the
>>> >>>>>>> partitioning
>>> >>>>>>> >>> job for graph processing.
>>> >>>>>>> >>>
>>> >>>>>>> >>> For other partitionings and with regard to our superstep API,
>>> Suraj's
>>> >>>>>>> idea
>>> >>>>>>> >>> of injecting a preprocessing superstep that partitions the
>>> stuff into
>>> >>>>>>> our
>>> >>>>>>> >>> messaging system is actually the best.
>>> >>>>>>> >>>
>>> >>>>>>> >>>
>>> >>>>>>> >>> 2013/3/6 Suraj Menon <surajsmenon@apache.org>
>>> >>>>>>> >>>
>>> >>>>>>> >>>> No, the partitions we write locally need not be sorted. Sorry
>>> for the
>>> >>>>>>> >>>> confusion. The Superstep injection is possible with Superstep
>>> API.
>>> >>>>>>> There
>>> >>>>>>> >>>> are few enhancements needed to make it simpler after I last
>>> worked on
>>> >>>>>>> it.
>>> >>>>>>> >>>> We can then look into partitioning superstep being executed
>>> before the
>>> >>>>>>> >>>> setup of first superstep of submitted job. I think it is
>>> feasible.
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. Yoon <
>>> edwardyoon@apache.org
>>> >>>>>>> >>>> >wrote:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject
>>> the
>>> >>>>>>> >>>> partitioning
>>> >>>>>>> >>>> > > superstep as the first superstep and use local memory?
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>> > Actually, I wanted to add something before calling
>>> BSP.setup()
>>> >>>>>>> method
>>> >>>>>>> >>>> > to avoid execute additional BSP job. But, in my opinion,
>>> current is
>>> >>>>>>> >>>> > enough. I think, we need to collect more experiences of
>>> input
>>> >>>>>>> >>>> > partitioning on large environments. I'll do.
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>> > BTW, I still don't know why it need to be Sorted?! MR-like?
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj Menon <
>>> >>>>>>> surajsmenon@apache.org>
>>> >>>>>>> >>>> > wrote:
>>> >>>>>>> >>>> > > Sorry, I am increasing the scope here to outside graph
>>> module.
>>> >>>>>>> When we
>>> >>>>>>> >>>> > have
>>> >>>>>>> >>>> > > spilling queue and sorted spilling queue, can we inject
>>> the
>>> >>>>>>> >>>> partitioning
>>> >>>>>>> >>>> > > superstep as the first superstep and use local memory?
>>> >>>>>>> >>>> > > Today we have partitioning job within a job and are
>>> creating two
>>> >>>>>>> copies
>>> >>>>>>> >>>> > of
>>> >>>>>>> >>>> > > data on HDFS. This could be really costly. Is it possible
>>> to
>>> >>>>>>> create or
>>> >>>>>>> >>>> > > redistribute the partitions on local memory and
>>> initialize the
>>> >>>>>>> record
>>> >>>>>>> >>>> > > reader there?
>>> >>>>>>> >>>> > > The user can run a separate job give in examples area to
>>> >>>>>>> explicitly
>>> >>>>>>> >>>> > > repartition the data on HDFS. The deployment question is
>>> how much
>>> >>>>>>> of
>>> >>>>>>> >>>> disk
>>> >>>>>>> >>>> > > space gets allocated for local memory usage? Would it be
>>> a safe
>>> >>>>>>> >>>> approach
>>> >>>>>>> >>>> > > with the limitations?
>>> >>>>>>> >>>> > >
>>> >>>>>>> >>>> > > -Suraj
>>> >>>>>>> >>>> > >
>>> >>>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas Jungblut
>>> >>>>>>> >>>> > > <thomas.jungblut@gmail.com>wrote:
>>> >>>>>>> >>>> > >
>>> >>>>>>> >>>> > >> yes. Once Suraj added merging of sorted files we can add
>>> this to
>>> >>>>>>> the
>>> >>>>>>> >>>> > >> partitioner pretty easily.
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> > >> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data really necessary
>>> to be
>>> >>>>>>> Sorted?
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Thomas Jungblut
>>> >>>>>>> >>>> > >> > <thomas.jungblut@gmail.com> wrote:
>>> >>>>>>> >>>> > >> > > Now I get how the partitioning works, obviously if
>>> you merge
>>> >>>>>>> n
>>> >>>>>>> >>>> > sorted
>>> >>>>>>> >>>> > >> > files
>>> >>>>>>> >>>> > >> > > by just appending to each other, this will result in
>>> totally
>>> >>>>>>> >>>> > unsorted
>>> >>>>>>> >>>> > >> > data
>>> >>>>>>> >>>> > >> > > ;-)
>>> >>>>>>> >>>> > >> > > Why didn't you solve this via messaging?
>>> >>>>>>> >>>> > >> > >
>>> >>>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut <thomas.jungblut@gmail.com
>>> >
>>> >>>>>>> >>>> > >> > >
>>> >>>>>>> >>>> > >> > >> Seems that they are not correctly sorted:
>>> >>>>>>> >>>> > >> > >>
>>> >>>>>>> >>>> > >> > >> vertexID: 50
>>> >>>>>>> >>>> > >> > >> vertexID: 52
>>> >>>>>>> >>>> > >> > >> vertexID: 54
>>> >>>>>>> >>>> > >> > >> vertexID: 56
>>> >>>>>>> >>>> > >> > >> vertexID: 58
>>> >>>>>>> >>>> > >> > >> vertexID: 61
>>> >>>>>>> >>>> > >> > >> ...
>>> >>>>>>> >>>> > >> > >> vertexID: 78
>>> >>>>>>> >>>> > >> > >> vertexID: 81
>>> >>>>>>> >>>> > >> > >> vertexID: 83
>>> >>>>>>> >>>> > >> > >> vertexID: 85
>>> >>>>>>> >>>> > >> > >> ...
>>> >>>>>>> >>>> > >> > >> vertexID: 94
>>> >>>>>>> >>>> > >> > >> vertexID: 96
>>> >>>>>>> >>>> > >> > >> vertexID: 98
>>> >>>>>>> >>>> > >> > >> vertexID: 1
>>> >>>>>>> >>>> > >> > >> vertexID: 10
>>> >>>>>>> >>>> > >> > >> vertexID: 12
>>> >>>>>>> >>>> > >> > >> vertexID: 14
>>> >>>>>>> >>>> > >> > >> vertexID: 16
>>> >>>>>>> >>>> > >> > >> vertexID: 18
>>> >>>>>>> >>>> > >> > >> vertexID: 21
>>> >>>>>>> >>>> > >> > >> vertexID: 23
>>> >>>>>>> >>>> > >> > >> vertexID: 25
>>> >>>>>>> >>>> > >> > >> vertexID: 27
>>> >>>>>>> >>>> > >> > >> vertexID: 29
>>> >>>>>>> >>>> > >> > >> vertexID: 3
>>> >>>>>>> >>>> > >> > >>
>>> >>>>>>> >>>> > >> > >> So this won't work then correctly...
>>> >>>>>>> >>>> > >> > >>
>>> >>>>>>> >>>> > >> > >>
>>> >>>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut <
>>> thomas.jungblut@gmail.com>
>>> >>>>>>> >>>> > >> > >>
>>> >>>>>>> >>>> > >> > >>> sure, have fun on your holidays.
>>> >>>>>>> >>>> > >> > >>>
>>> >>>>>>> >>>> > >> > >>>
>>> >>>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon <edwardyoon@apache.org>
>>> >>>>>>> >>>> > >> > >>>
>>> >>>>>>> >>>> > >> > >>>> Sure, but if you can fix quickly, please do.
>>> March 1 is
>>> >>>>>>> >>>> > holiday[1]
>>> >>>>>>> >>>> > >> so
>>> >>>>>>> >>>> > >> > >>>> I'll appear next week.
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> > >>>> 1.
>>> >>>>>>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM, Thomas Jungblut
>>> >>>>>>> >>>> > >> > >>>> <thomas.jungblut@gmail.com> wrote:
>>> >>>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the file, didn't
>>> observe if all
>>> >>>>>>> >>>> items
>>> >>>>>>> >>>> > >> were
>>> >>>>>>> >>>> > >> > >>>> added.
>>> >>>>>>> >>>> > >> > >>>> > As far as I remember, I copy/pasted the logic
>>> of the ID
>>> >>>>>>> into
>>> >>>>>>> >>>> > the
>>> >>>>>>> >>>> > >> > >>>> fastgen,
>>> >>>>>>> >>>> > >> > >>>> > want to have a look into it?
>>> >>>>>>> >>>> > >> > >>>> >
>>> >>>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon <edwardyoon@apache.org
>>> >
>>> >>>>>>> >>>> > >> > >>>> >
>>> >>>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastgen, when generate
>>> adjacency
>>> >>>>>>> >>>> matrix
>>> >>>>>>> >>>> > >> into
>>> >>>>>>> >>>> > >> > >>>> >> multiple files.
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 PM, Thomas
>>> Jungblut
>>> >>>>>>> >>>> > >> > >>>> >> <thomas.jungblut@gmail.com> wrote:
>>> >>>>>>> >>>> > >> > >>>> >> > You have two files, are they partitioned
>>> correctly?
>>> >>>>>>> >>>> > >> > >>>> >> >
>>> >>>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon <
>>> edwardyoon@apache.org>
>>> >>>>>>> >>>> > >> > >>>> >> >
>>> >>>>>>> >>>> > >> > >>>> >> >> It looks like a bug.
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls
>>> -al
>>> >>>>>>> >>>> > >> /tmp/randomgraph/
>>> >>>>>>> >>>> > >> > >>>> >> >> total 44
>>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x  3 edward edward  4096  2월 28
>>> 18:03 .
>>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root   root   20480  2월 28
>>> 18:04 ..
>>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx  1 edward edward  2243  2월 28
>>> 18:01
>>> >>>>>>> part-00000
>>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r--  1 edward edward    28  2월 28
>>> 18:01
>>> >>>>>>> >>>> > .part-00000.crc
>>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx  1 edward edward  2251  2월 28
>>> 18:01
>>> >>>>>>> part-00001
>>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r--  1 edward edward    28  2월 28
>>> 18:01
>>> >>>>>>> >>>> > .part-00001.crc
>>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x  2 edward edward  4096  2월 28
>>> 18:03
>>> >>>>>>> partitions
>>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$ ls
>>> -al
>>> >>>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/
>>> >>>>>>> >>>> > >> > >>>> >> >> total 24
>>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edward 4096  2월 28
>>> 18:03 .
>>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edward 4096  2월 28
>>> 18:03 ..
>>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2932  2월 28 18:03
>>> >>>>>>> part-00000
>>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward   32  2월 28 18:03
>>> >>>>>>> >>>> > .part-00000.crc
>>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edward 2955  2월 28 18:03
>>> >>>>>>> part-00001
>>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edward   32  2월 28 18:03
>>> >>>>>>> >>>> > .part-00001.crc
>>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax:~/workspace/hama-trunk$
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5:27 PM, Edward <
>>> >>>>>>> >>>> edward@udanax.org
>>> >>>>>>> >>>> > >
>>> >>>>>>> >>>> > >> > wrote:
>>> >>>>>>> >>>> > >> > >>>> >> >> > yes i'll check again
>>> >>>>>>> >>>> > >> > >>>> >> >> >
>>> >>>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone
>>> >>>>>>> >>>> > >> > >>>> >> >> >
>>> >>>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:18 PM, Thomas
>>> Jungblut <
>>> >>>>>>> >>>> > >> > >>>> >> thomas.jungblut@gmail.com>
>>> >>>>>>> >>>> > >> > >>>> >> >> wrote:
>>> >>>>>>> >>>> > >> > >>>> >> >> >
>>> >>>>>>> >>>> > >> > >>>> >> >> >> Can you verify an observation for me
>>> please?
>>> >>>>>>> >>>> > >> > >>>> >> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> >> 2 files are created from fastgen,
>>> part-00000 and
>>> >>>>>>> >>>> > >> part-00001,
>>> >>>>>>> >>>> > >> > >>>> both
>>> >>>>>>> >>>> > >> > >>>> >> ~2.2kb
>>> >>>>>>> >>>> > >> > >>>> >> >> >> sized.
>>> >>>>>>> >>>> > >> > >>>> >> >> >> In the below partition directory, there
>>> is only a
>>> >>>>>>> >>>> single
>>> >>>>>>> >>>> > >> > 5.56kb
>>> >>>>>>> >>>> > >> > >>>> file.
>>> >>>>>>> >>>> > >> > >>>> >> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the partitioner to
>>> write a
>>> >>>>>>> single
>>> >>>>>>> >>>> > file
>>> >>>>>>> >>>> > >> if
>>> >>>>>>> >>>> > >> > you
>>> >>>>>>> >>>> > >> > >>>> >> >> configured
>>> >>>>>>> >>>> > >> > >>>> >> >> >> two?
>>> >>>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a two files, strange
>>> huh?
>>> >>>>>>> >>>> > >> > >>>> >> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas Jungblut <
>>> >>>>>>> thomas.jungblut@gmail.com>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into it.
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 /tmp/randomgraph 1
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> pagerank /tmp/randomgraph /tmp/pageout
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> did work for me the last time I
>>> profiled, maybe
>>> >>>>>>> the
>>> >>>>>>> >>>> > >> > >>>> partitioning
>>> >>>>>>> >>>> > >> > >>>> >> >> doesn't
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> partition correctly with the input or
>>> something
>>> >>>>>>> else.
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. Yoon <
>>> edwardyoon@apache.org
>>> >>>>>>> >
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems not work for graph
>>> examples.
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>>> >>>>>>> :~/workspace/hama-trunk$
>>> >>>>>>> >>>> > >> bin/hama
>>> >>>>>>> >>>> > >> > jar
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar gen
>>> >>>>>>> >>>> > >> > fastgen
>>> >>>>>>> >>>> > >> > >>>> 100 10
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WARN
>>> util.NativeCodeLoader:
>>> >>>>>>> Unable
>>> >>>>>>> >>>> > to
>>> >>>>>>> >>>> > >> > load
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your
>>> platform...
>>> >>>>>>> using
>>> >>>>>>> >>>> > >> > builtin-java
>>> >>>>>>> >>>> > >> > >>>> >> classes
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Running
>>> >>>>>>> >>>> job:
>>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 INFO
>>> bsp.LocalBSPRunner:
>>> >>>>>>> Setting
>>> >>>>>>> >>>> up
>>> >>>>>>> >>>> > a
>>> >>>>>>> >>>> > >> new
>>> >>>>>>> >>>> > >> > >>>> barrier
>>> >>>>>>> >>>> > >> > >>>> >> >> for 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Current
>>> >>>>>>> >>>> > >> supersteps
>>> >>>>>>> >>>> > >> > >>>> >> number: 0
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient: The
>>> >>>>>>> total
>>> >>>>>>> >>>> > number
>>> >>>>>>> >>>> > >> > of
>>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 0
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Counters: 3
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> org.apache.hama.bsp.JobInProgress$JobCounter
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > SUPERSTEPS=0
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=100
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.212 seconds
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>>> >>>>>>> :~/workspace/hama-trunk$
>>> >>>>>>> >>>> > >> bin/hama
>>> >>>>>>> >>>> > >> > jar
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> examples/target/hama-examples-0.7.0-SNAPSHOT
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> hama-examples-0.7.0-SNAPSHOT.jar
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> edward@edward-virtualBox
>>> >>>>>>> :~/workspace/hama-trunk$
>>> >>>>>>> >>>> > >> bin/hama
>>> >>>>>>> >>>> > >> > jar
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar
>>> >>>>>>> >>>> > pagerank
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph /tmp/pageour
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WARN
>>> util.NativeCodeLoader:
>>> >>>>>>> Unable
>>> >>>>>>> >>>> > to
>>> >>>>>>> >>>> > >> > load
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop library for your
>>> platform...
>>> >>>>>>> using
>>> >>>>>>> >>>> > >> > builtin-java
>>> >>>>>>> >>>> > >> > >>>> >> classes
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO
>>> bsp.FileInputFormat:
>>> >>>>>>> Total
>>> >>>>>>> >>>> > input
>>> >>>>>>> >>>> > >> > paths
>>> >>>>>>> >>>> > >> > >>>> to
>>> >>>>>>> >>>> > >> > >>>> >> >> process
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 INFO
>>> bsp.FileInputFormat:
>>> >>>>>>> Total
>>> >>>>>>> >>>> > input
>>> >>>>>>> >>>> > >> > paths
>>> >>>>>>> >>>> > >> > >>>> to
>>> >>>>>>> >>>> > >> > >>>> >> >> process
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Running
>>> >>>>>>> >>>> job:
>>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 INFO
>>> bsp.LocalBSPRunner:
>>> >>>>>>> Setting
>>> >>>>>>> >>>> up
>>> >>>>>>> >>>> > a
>>> >>>>>>> >>>> > >> new
>>> >>>>>>> >>>> > >> > >>>> barrier
>>> >>>>>>> >>>> > >> > >>>> >> >> for 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Current
>>> >>>>>>> >>>> > >> supersteps
>>> >>>>>>> >>>> > >> > >>>> >> number: 1
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient: The
>>> >>>>>>> total
>>> >>>>>>> >>>> > number
>>> >>>>>>> >>>> > >> > of
>>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 1
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Counters: 6
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> org.apache.hama.bsp.JobInProgress$JobCounter
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > SUPERSTEPS=1
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > SUPERSTEP_SUM=4
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> IO_BYTES_READ=4332
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=14
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=100
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.FileInputFormat:
>>> >>>>>>> Total
>>> >>>>>>> >>>> > input
>>> >>>>>>> >>>> > >> > paths
>>> >>>>>>> >>>> > >> > >>>> to
>>> >>>>>>> >>>> > >> > >>>> >> >> process
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.BSPJobClient:
>>> >>>>>>> Running
>>> >>>>>>> >>>> job:
>>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> bsp.LocalBSPRunner:
>>> >>>>>>> Setting
>>> >>>>>>> >>>> up
>>> >>>>>>> >>>> > a
>>> >>>>>>> >>>> > >> new
>>> >>>>>>> >>>> > >> > >>>> barrier
>>> >>>>>>> >>>> > >> > >>>> >> >> for 2
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks!
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> graph.GraphJobRunner: 50
>>> >>>>>>> >>>> > vertices
>>> >>>>>>> >>>> > >> > are
>>> >>>>>>> >>>> > >> > >>>> loaded
>>> >>>>>>> >>>> > >> > >>>> >> >> into
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:1
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 INFO
>>> graph.GraphJobRunner: 50
>>> >>>>>>> >>>> > vertices
>>> >>>>>>> >>>> > >> > are
>>> >>>>>>> >>>> > >> > >>>> loaded
>>> >>>>>>> >>>> > >> > >>>> >> >> into
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:0
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ERROR
>>> bsp.LocalBSPRunner:
>>> >>>>>>> >>>> Exception
>>> >>>>>>> >>>> > >> > during
>>> >>>>>>> >>>> > >> > >>>> BSP
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> execution!
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> java.lang.IllegalArgumentException:
>>> Messages
>>> >>>>>>> must
>>> >>>>>>> >>>> > never
>>> >>>>>>> >>>> > >> be
>>> >>>>>>> >>>> > >> > >>>> behind
>>> >>>>>>> >>>> > >> > >>>> >> the
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Current Message ID: 1
>>> vs. 50
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>>
>>> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> >
>>> >>>>>>>
>>> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>>
>>> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> >
>>> >>>>>>>
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>>
>>> >>>>>>>
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>>
>>> >>>>>>>
>>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> >
>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>>
>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> >
>>> >>>>>>>
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> >
>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>>
>>> java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>>
>>> >>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>>
>>> >>>>>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>        at
>>> java.lang.Thread.run(Thread.java:722)
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> --
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward J. Yoon
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >> >>>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >> >> --
>>> >>>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Yoon
>>> >>>>>>> >>>> > >> > >>>> >> >> @eddieyoon
>>> >>>>>>> >>>> > >> > >>>> >> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>> >> --
>>> >>>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon
>>> >>>>>>> >>>> > >> > >>>> >> @eddieyoon
>>> >>>>>>> >>>> > >> > >>>> >>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> > >>>> --
>>> >>>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon
>>> >>>>>>> >>>> > >> > >>>> @eddieyoon
>>> >>>>>>> >>>> > >> > >>>>
>>> >>>>>>> >>>> > >> > >>>
>>> >>>>>>> >>>> > >> > >>>
>>> >>>>>>> >>>> > >> > >>
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >> > --
>>> >>>>>>> >>>> > >> > Best Regards, Edward J. Yoon
>>> >>>>>>> >>>> > >> > @eddieyoon
>>> >>>>>>> >>>> > >> >
>>> >>>>>>> >>>> > >>
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>> > --
>>> >>>>>>> >>>> > Best Regards, Edward J. Yoon
>>> >>>>>>> >>>> > @eddieyoon
>>> >>>>>>> >>>> >
>>> >>>>>>> >>>>
>>> >>>>>>> >>
>>> >>>>>>> >>
>>> >>>>>>> >>
>>> >>>>>>> >> --
>>> >>>>>>> >> Best Regards, Edward J. Yoon
>>> >>>>>>> >> @eddieyoon
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > --
>>> >>>>>>> > Best Regards, Edward J. Yoon
>>> >>>>>>> > @eddieyoon
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --
>>> >>>>>>> Best Regards, Edward J. Yoon
>>> >>>>>>> @eddieyoon
>>> >>>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Best Regards, Edward J. Yoon
>>> >>>>> @eddieyoon
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Best Regards, Edward J. Yoon
>>> >>>> @eddieyoon
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Best Regards, Edward J. Yoon
>>> >>> @eddieyoon
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards, Edward J. Yoon
>>> >> @eddieyoon
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Edward J. Yoon
>>> > @eddieyoon
>>>
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> @eddieyoon
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message