Return-Path: X-Original-To: apmail-hama-dev-archive@www.apache.org Delivered-To: apmail-hama-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 68F93D6F2 for ; Thu, 14 Mar 2013 21:05:35 +0000 (UTC) Received: (qmail 42544 invoked by uid 500); 14 Mar 2013 21:05:35 -0000 Delivered-To: apmail-hama-dev-archive@hama.apache.org Received: (qmail 42498 invoked by uid 500); 14 Mar 2013 21:05:35 -0000 Mailing-List: contact dev-help@hama.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hama.apache.org Delivered-To: mailing list dev@hama.apache.org Received: (qmail 42488 invoked by uid 99); 14 Mar 2013 21:05:35 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Mar 2013 21:05:35 +0000 Received: from localhost (HELO mail-ie0-f173.google.com) (127.0.0.1) (smtp-auth username edwardyoon, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Mar 2013 21:05:34 +0000 Received: by mail-ie0-f173.google.com with SMTP id 9so3627863iec.4 for ; Thu, 14 Mar 2013 14:05:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:content-transfer-encoding :x-gm-message-state; bh=EU0S8BS2FdxrJ4L3DigTXiXhM6LI/RAt0FQZge3oviQ=; b=izBUopZ0I5pF0qjkEad6yv6CCtyGbtRl7YiDb7HMd9/TQcLyTH9LR9o0M2cqEqftq2 TWUFc8C9/RV4pyjgeVZQJ12SdIs7qOcPeQBFJrL9Vbrk0nmJrd8rwX8cGezWrffrq0sI PweZ8f0vUc0GSmOPkbzrJKOdWkUkzzG398hXGlClCicFLSUZ5d+0DUkGo2bQeo59LYHF StTXkQ+kUiP9ayzh2c2YXwOoIcR5nMYn7ulbhjWcMqwYsCi7c8ocV3xWU1uOE1I5nNFL BxenuxZdFHMPlNU/Z2e752Q+4RJsYDdd5BNZrAWRyBLQDYzlUNZCMrFBKD3caBWdm4qz BMEw== MIME-Version: 1.0 X-Received: by 10.50.178.105 with SMTP id cx9mr21466957igc.111.1363295133811; Thu, 14 Mar 2013 14:05:33 -0700 (PDT) Received: by 10.64.33.140 with HTTP; Thu, 14 Mar 2013 14:05:33 -0700 (PDT) In-Reply-To: References: Date: Fri, 15 Mar 2013 06:05:33 +0900 Message-ID: Subject: Re: Error with fastgen input From: "Edward J. Yoon" To: dev@hama.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQm4wGJ8eDDfUp2Ej2r+gjc97zisiuQUVVdF8TvJGAPa3S6WntYIa5wIk3DtUfCNFOvP2HkI Hmm Okay. The problem looks come from review culture. On Fri, Mar 15, 2013 at 12:54 AM, Suraj Menon wrot= e: >> It can only be answered by patches? > > The answer is partly yes. As an example, please refer to the conversation > we had in https://issues.apache.org/jira/browse/HAMA-559 > I think between me and Thomas we tried and went to and forth between > atleast three to four designs before we finalized on one. (Mind you the > read performance for spilling queue was fixed later.) > >> can we discuss about our plan for vertices first? > > The design we(I thought including you) are contemplating now is to do a > join of 2 sorted entities (Vertices and Messages). This is implied in > HAMA-704 final patch, that has your +1(?). With synchronized communicatio= n > and sorted queues, you did suggest somewhere that the performance was > slower(which was expected). So to speed up with scalability, we should do > async communication and spilled sorted queue. Now this needs refactoring > everything we know in messaging code. So patch by patch we would have to > reach there. I don't think there would be a redo. Because most of these a= re > building blocks for other applications waiting for these changes. > > Thanks, > Suraj > > > > On Thu, Mar 14, 2013 at 10:01 AM, Edward J. Yoon w= rote: > >> This is pure question. >> >> Before we discussing about below issues, can we discuss about our plan >> for vertices first? Do I need to wait to see if messaging system can >> be used to sort partitioned data by vertex comparator until all >> Spilling queue related issues are fixed? It can only be answered by >> patches? >> >> On Thu, Mar 14, 2013 at 8:40 PM, Suraj Menon >> wrote: >> > Going in line with the latest topic of the conversation. >> > Nothing is closed here and the JIRA's were already created for the who= le >> > thing to come in place: >> > >> > HAMA-644 >> > HAMA-490 >> > HAMA-722 >> > HAMA-728 >> > HAMA-707 >> > HAMA-728 >> > >> > The JIRA's above are directly or indirectly affected during core >> > refactoring. >> > >> > -Suraj >> > >> > >> > On Thu, Mar 14, 2013 at 7:03 AM, Edward J. Yoon > >wrote: >> > >> >> P.S., These comments are never helpful in developing community. >> >> >> >> "before you run riot on all along the codebase, Suraj ist currently >> working >> >> on that stuff- don't make it more difficult for him rebasing all his >> >> patches the whole time. >> >> He has the plan so that we made to make the stuff working, his part i= s >> >> currently missing. So don't try to muddle arround there, it will make >> this >> >> take longer than already needed." >> >> >> >> On Thu, Mar 14, 2013 at 7:57 PM, Edward J. Yoon >> >> wrote: >> >> > In my opinion, the our best action is - 1) explain the plans, edit >> >> > together on Wiki, and then 2) break-down implementation tasks as sm= all >> >> > as possible so that available people can try it in parallel. Then, = you >> >> > can use available people. Do you remember, I asked you to write dow= n >> >> > your plan here? - http://wiki.apache.org/hama/SpillingQueue If you >> >> > have some time, Please do for me. I'll help you in my free time. >> >> > >> >> > Regarding branches, maybe we all are not familiar with online >> >> > collaboration (or don't want to collaborate anymore). If we want to >> >> > walk own ways, why we need to be in here together? >> >> > >> >> > On Thu, Mar 14, 2013 at 7:13 PM, Suraj Menon >> >> wrote: >> >> >> Three points: >> >> >> >> >> >> Firstly, apologies because partly this conversation emanates from = the >> >> delay >> >> >> in providing the set of patches. I was not able to slice as much >> time I >> >> was >> >> >> hoping. >> >> >> >> >> >> Second, I think I/we can work on a separate branches. Since most o= f >> >> these >> >> >> concerns could only be answered by future patches, a decision coul= d >> be >> >> made >> >> >> then. We can decide if svn revert is needed during the process on >> trunk. >> >> >> (This is a general comment and not related to particular JIRA) >> >> >> >> >> >> Third, Please feel free to slice a release if it is really importa= nt. >> >> >> >> >> >> Thanks, >> >> >> Suraj >> >> >> >> >> >> On Thu, Mar 14, 2013 at 5:39 AM, Edward J. Yoon < >> edwardyoon@apache.org >> >> >wrote: >> >> >> >> >> >>> To reduce arguing, I'm appending my opinions. >> >> >>> >> >> >>> In HAMA-704, I wanted to remove only message map to reduce memory >> >> >>> consumption. I still don't want to talk about disk-based vertices >> and >> >> >>> Spilling Queue at the moment. With this, I wanted to release 0.6.= 1 >> >> >>> 'partitioning issue fixed and quick executable examples' version >> ASAP. >> >> >>> That's why I scheduled Spilling Queue in 0.7 roadmap. >> >> >>> >> >> >>> As you can see, issues are happening one right after another. I >> don't >> >> >>> think we have to clean all never-ending issues. We can improve >> >> >>> step-by-step. >> >> >>> >> >> >>> 1. http://wiki.apache.org/hama/RoadMap >> >> >>> >> >> >>> On Thu, Mar 14, 2013 at 6:22 PM, Edward J. Yoon < >> edwardyoon@apache.org >> >> > >> >> >>> wrote: >> >> >>> > Typos ;) >> >> >>> > >> >> >>> >> except YARN integration tasks. If you leave here, I have to ta= ke >> >> cover >> >> >>> >> YARN tasks. Should I wait someone? Am I touching core module >> >> >>> > >> >> >>> > I have to cover YARN tasks instead of you. >> >> >>> > >> >> >>> > On Thu, Mar 14, 2013 at 6:12 PM, Edward J. Yoon < >> >> edwardyoon@apache.org> >> >> >>> wrote: >> >> >>> >> Hmm, here's my opinions: >> >> >>> >> >> >> >>> >> As you know, we have a problem of lack of team members and >> >> >>> >> contributors. So we should break down every tasks as small as >> >> >>> >> possible. Our best action is improving step-by-step. And every >> >> >>> >> Hama-x.x.x should run well even though it's a baby cart level. >> >> >>> >> >> >> >>> >> And, Tech should be developed under the necessity. So I think = we >> >> need >> >> >>> >> to cut release as often as possible. Therefore I volunteered t= o >> >> manage >> >> >>> >> release. Actually, I was wanted to work only on QA (quality >> >> assurance) >> >> >>> >> related tasks because yours code is better than me and I have = a >> >> >>> >> cluster. >> >> >>> >> >> >> >>> >> However, we are currently not doing like that. I guess there a= re >> >> many >> >> >>> >> reasons. We're all not a full-time open sourcer (except me). >> >> >>> >> >> >> >>> >>> You have 23 issues assigned. Why do you need to work on that= ? >> >> >>> >> >> >> >>> >> I don't know what you mean exactly. But 23 issues are almost >> >> examples >> >> >>> >> except YARN integration tasks. If you leave here, I have to ta= ke >> >> cover >> >> >>> >> YARN tasks. Should I wait someone? Am I touching core module >> >> >>> >> aggressively? >> >> >>> >> >> >> >>> >>> Otherwise Suraj and I branch that issues away and you can pla= y >> >> >>> arround.l in >> >> >>> >>> trunk how you like. >> >> >>> >> >> >> >>> >> I also don't know what you mean exactly but if you want, Pleas= e >> do. >> >> >>> >> >> >> >>> >> By the way, can you answer about this question - Is it really >> >> >>> >> technical conflicts? or emotional conflicts? >> >> >>> >> >> >> >>> >> On Thu, Mar 14, 2013 at 5:32 PM, Thomas Jungblut >> >> >>> >> wrote: >> >> >>> >>> You have 23 issues assigned. Why do you need to work on that= ? >> >> >>> >>> Otherwise Suraj and I branch that issues away and you can pla= y >> >> >>> arround.l in >> >> >>> >>> trunk how you like. >> >> >>> >>> Am 14.03.2013 09:04 schrieb "Edward J. Yoon" < >> >> edwardyoon@apache.org>: >> >> >>> >>> >> >> >>> >>>> P.S., Please don't say like that. >> >> >>> >>>> >> >> >>> >>>> No decisions made yet. And if someone have a question or mis= sed >> >> >>> >>>> something, you have to try to explain here. Because this is = a >> open >> >> >>> >>>> source. Anyone can't say "don't touch trunk bc I'm working o= n >> it". >> >> >>> >>>> >> >> >>> >>>> On Thu, Mar 14, 2013 at 4:37 PM, Edward J. Yoon < >> >> >>> edwardyoon@apache.org> >> >> >>> >>>> wrote: >> >> >>> >>>> > Sorry for my quick and dirty style small patches. >> >> >>> >>>> > >> >> >>> >>>> > However, we should work together in parallel. Please share >> here >> >> if >> >> >>> >>>> > there are some progresses. >> >> >>> >>>> > >> >> >>> >>>> > On Thu, Mar 14, 2013 at 3:46 PM, Thomas Jungblut >> >> >>> >>>> > wrote: >> >> >>> >>>> >> Hi Edward, >> >> >>> >>>> >> >> >> >>> >>>> >> before you run riot on all along the codebase, Suraj ist >> >> currently >> >> >>> >>>> working >> >> >>> >>>> >> on that stuff- don't make it more difficult for him rebas= ing >> >> all >> >> >>> his >> >> >>> >>>> >> patches the whole time. >> >> >>> >>>> >> He has the plan so that we made to make the stuff working= , >> his >> >> >>> part is >> >> >>> >>>> >> currently missing. So don't try to muddle arround there, = it >> >> will >> >> >>> make >> >> >>> >>>> this >> >> >>> >>>> >> take longer than already needed. >> >> >>> >>>> >> >> >> >>> >>>> >> >> >> >>> >>>> >> >> >> >>> >>>> >> 2013/3/14 Edward J. Yoon >> >> >>> >>>> >> >> >> >>> >>>> >>> Personally, I would like to solve this issue by touching >> >> >>> >>>> >>> DiskVerticesInfo. If we write sorted sub-sets of vertice= s >> into >> >> >>> >>>> >>> multiple files, we can avoid huge memory consumption. >> >> >>> >>>> >>> >> >> >>> >>>> >>> If we want to sort partitioned data using messaging syst= em, >> >> idea >> >> >>> >>>> >>> should be collected. >> >> >>> >>>> >>> >> >> >>> >>>> >>> On Thu, Mar 14, 2013 at 10:31 AM, Edward J. Yoon < >> >> >>> >>>> edwardyoon@apache.org> >> >> >>> >>>> >>> wrote: >> >> >>> >>>> >>> > Oh, now I get how iterate() works. HAMA-704 is nicely >> >> written. >> >> >>> >>>> >>> > >> >> >>> >>>> >>> > On Thu, Mar 14, 2013 at 12:02 AM, Edward J. Yoon < >> >> >>> >>>> edwardyoon@apache.org> >> >> >>> >>>> >>> wrote: >> >> >>> >>>> >>> >> I'm reading changes of HAMA-704 again. As a result of >> >> adding >> >> >>> >>>> >>> >> DiskVerticesInfo, vertices list is needed to be sorte= d. >> I'm >> >> >>> not sure >> >> >>> >>>> >>> >> but I think this approach will bring more disadvantag= es >> >> than >> >> >>> >>>> >>> >> advantages. >> >> >>> >>>> >>> >> >> >> >>> >>>> >>> >> On Wed, Mar 13, 2013 at 11:09 PM, Edward J. Yoon < >> >> >>> >>>> edwardyoon@apache.org> >> >> >>> >>>> >>> wrote: >> >> >>> >>>> >>> >>>>>> in loadVertices? Maybe consider feature for coupl= ing >> >> >>> storage in >> >> >>> >>>> >>> user space >> >> >>> >>>> >>> >>>>>> with BSP Messaging[HAMA-734] can avoid double rea= ds >> and >> >> >>> writes. >> >> >>> >>>> >>> This way >> >> >>> >>>> >>> >>>>>> partitioned or non-partitioned by partitioner, ca= n >> keep >> >> >>> vertices >> >> >>> >>>> >>> sorted >> >> >>> >>>> >>> >>>>>> with a single read and single write on every peer= . >> >> >>> >>>> >>> >>> >> >> >>> >>>> >>> >>> And, as I commented JIRA ticket, I think we can't us= e >> >> >>> messaging >> >> >>> >>>> system >> >> >>> >>>> >>> >>> for sorting vertices within partition files. >> >> >>> >>>> >>> >>> >> >> >>> >>>> >>> >>> On Wed, Mar 13, 2013 at 11:00 PM, Edward J. Yoon < >> >> >>> >>>> >>> edwardyoon@apache.org> wrote: >> >> >>> >>>> >>> >>>> P.S., (number of splits =3D number of partitions) i= s >> really >> >> >>> confuse >> >> >>> >>>> to >> >> >>> >>>> >>> >>>> me. Even though blocks number is equal to desired >> tasks >> >> >>> number, >> >> >>> >>>> data >> >> >>> >>>> >>> >>>> should be re-partitioned again. >> >> >>> >>>> >>> >>>> >> >> >>> >>>> >>> >>>> On Wed, Mar 13, 2013 at 10:36 PM, Edward J. Yoon < >> >> >>> >>>> >>> edwardyoon@apache.org> wrote: >> >> >>> >>>> >>> >>>>> Indeed. If there are already partitioned input fil= es >> >> >>> (unsorted) >> >> >>> >>>> and >> >> >>> >>>> >>> so >> >> >>> >>>> >>> >>>>> user want to skip pre-partitioning phase, it shoul= d >> be >> >> >>> handled in >> >> >>> >>>> >>> >>>>> GraphJobRunner BSP program. Actually, I still don'= t >> >> know why >> >> >>> >>>> >>> >>>>> re-partitioned files need to be Sorted. It's only >> about >> >> >>> >>>> >>> >>>>> GraphJobRunner. >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>>> partitioning. (This is outside the scope of graph= s. >> We >> >> can >> >> >>> have >> >> >>> >>>> a >> >> >>> >>>> >>> dedicated >> >> >>> >>>> >>> >>>>>> partitioning superstep for graph applications). >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> Sorry. I don't understand exactly yet. Do you mean >> just >> >> a >> >> >>> >>>> >>> partitioning >> >> >>> >>>> >>> >>>>> job based on superstep API? >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> By default, 100 tasks will be assigned for >> partitioning >> >> job. >> >> >>> >>>> >>> >>>>> Partitioning job will create 1,000 partitions. Thu= s, >> we >> >> can >> >> >>> >>>> execute >> >> >>> >>>> >>> >>>>> the Graph job with 1,000 tasks. >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> Let's assume that a input sequence file is 20GB (1= 00 >> >> >>> blocks). If >> >> >>> >>>> I >> >> >>> >>>> >>> >>>>> want to run with 1,000 tasks, what happens? >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> On Wed, Mar 13, 2013 at 9:49 PM, Suraj Menon < >> >> >>> >>>> surajsmenon@apache.org> >> >> >>> >>>> >>> wrote: >> >> >>> >>>> >>> >>>>>> I am responding on this thread because of better >> >> >>> continuity for >> >> >>> >>>> >>> >>>>>> conversation. We cannot expect the partitions to = be >> >> sorted >> >> >>> every >> >> >>> >>>> >>> time. When >> >> >>> >>>> >>> >>>>>> the number of splits =3D number of partitions and >> >> >>> partitioning is >> >> >>> >>>> >>> switched >> >> >>> >>>> >>> >>>>>> off by user[HAMA-561], the partitions would not b= e >> >> sorted. >> >> >>> Can >> >> >>> >>>> we >> >> >>> >>>> >>> do this >> >> >>> >>>> >>> >>>>>> in loadVertices? Maybe consider feature for coupl= ing >> >> >>> storage in >> >> >>> >>>> >>> user space >> >> >>> >>>> >>> >>>>>> with BSP Messaging[HAMA-734] can avoid double rea= ds >> and >> >> >>> writes. >> >> >>> >>>> >>> This way >> >> >>> >>>> >>> >>>>>> partitioned or non-partitioned by partitioner, ca= n >> keep >> >> >>> vertices >> >> >>> >>>> >>> sorted >> >> >>> >>>> >>> >>>>>> with a single read and single write on every peer= . >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> Just clearing confusion if any regarding superste= p >> >> >>> injection for >> >> >>> >>>> >>> >>>>>> partitioning. (This is outside the scope of graph= s. >> We >> >> can >> >> >>> have >> >> >>> >>>> a >> >> >>> >>>> >>> dedicated >> >> >>> >>>> >>> >>>>>> partitioning superstep for graph applications). >> >> >>> >>>> >>> >>>>>> Say there are x splits and y number of tasks >> >> configured by >> >> >>> user. >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> if x > y >> >> >>> >>>> >>> >>>>>> The y tasks are scheduled with x of them having >> each of >> >> >>> the x >> >> >>> >>>> >>> splits and >> >> >>> >>>> >>> >>>>>> the remaining with no resource local to them. The= n >> the >> >> >>> >>>> partitioning >> >> >>> >>>> >>> >>>>>> superstep redistributes the partitions among them= to >> >> create >> >> >>> >>>> local >> >> >>> >>>> >>> >>>>>> partitions. Now the question is can we >> re-initialize a >> >> >>> peer's >> >> >>> >>>> input >> >> >>> >>>> >>> based >> >> >>> >>>> >>> >>>>>> on this new local part of partition? >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> if y > x >> >> >>> >>>> >>> >>>>>> works as it works today. >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> Just putting my points in brainstorming. >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> -Suraj >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>> On Mon, Mar 11, 2013 at 7:39 AM, Edward J. Yoon < >> >> >>> >>>> >>> edwardyoon@apache.org>wrote: >> >> >>> >>>> >>> >>>>>> >> >> >>> >>>> >>> >>>>>>> I just filed here >> >> >>> >>>> https://issues.apache.org/jira/browse/HAMA-744 >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >>>>>>> On Mon, Mar 11, 2013 at 7:35 PM, Edward J. Yoon = < >> >> >>> >>>> >>> edwardyoon@apache.org> >> >> >>> >>>> >>> >>>>>>> wrote: >> >> >>> >>>> >>> >>>>>>> > Additionally, >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> >> spilling queue and sorted spilling queue, can= we >> >> >>> inject the >> >> >>> >>>> >>> partitioning >> >> >>> >>>> >>> >>>>>>> >> superstep as the first superstep and use loca= l >> >> memory? >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> > Can we execute different number of tasks per >> >> superstep? >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> > On Mon, Mar 11, 2013 at 6:56 PM, Edward J. Yoo= n < >> >> >>> >>>> >>> edwardyoon@apache.org> >> >> >>> >>>> >>> >>>>>>> wrote: >> >> >>> >>>> >>> >>>>>>> >>> For graph processing, the partitioned files >> that >> >> >>> result >> >> >>> >>>> from >> >> >>> >>>> >>> the >> >> >>> >>>> >>> >>>>>>> >>> partitioning job must be sorted. Currently o= nly >> >> the >> >> >>> >>>> partition >> >> >>> >>>> >>> files in >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >> I see. >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >>> For other partitionings and with regard to o= ur >> >> >>> superstep >> >> >>> >>>> API, >> >> >>> >>>> >>> Suraj's >> >> >>> >>>> >>> >>>>>>> idea >> >> >>> >>>> >>> >>>>>>> >>> of injecting a preprocessing superstep that >> >> >>> partitions the >> >> >>> >>>> >>> stuff into >> >> >>> >>>> >>> >>>>>>> our >> >> >>> >>>> >>> >>>>>>> >>> messaging system is actually the best. >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >> BTW, if some garbage objects can be accumulat= ed >> in >> >> >>> >>>> partitioning >> >> >>> >>>> >>> step, >> >> >>> >>>> >>> >>>>>>> >> separated partitioning job may not be bad ide= a. >> Is >> >> >>> there >> >> >>> >>>> some >> >> >>> >>>> >>> special >> >> >>> >>>> >>> >>>>>>> >> reason? >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >> On Wed, Mar 6, 2013 at 6:15 PM, Thomas Jungbl= ut >> >> >>> >>>> >>> >>>>>>> >> wrote: >> >> >>> >>>> >>> >>>>>>> >>> For graph processing, the partitioned files >> that >> >> >>> result >> >> >>> >>>> from >> >> >>> >>>> >>> the >> >> >>> >>>> >>> >>>>>>> >>> partitioning job must be sorted. Currently o= nly >> >> the >> >> >>> >>>> partition >> >> >>> >>>> >>> files in >> >> >>> >>>> >>> >>>>>>> >>> itself are sorted, thus more tasks result in >> not >> >> >>> sorted >> >> >>> >>>> data >> >> >>> >>>> >>> in the >> >> >>> >>>> >>> >>>>>>> >>> completed file. This only applies for the gr= aph >> >> >>> processing >> >> >>> >>>> >>> package. >> >> >>> >>>> >>> >>>>>>> >>> So as Suraj told, it would be much more >> simpler to >> >> >>> solve >> >> >>> >>>> this >> >> >>> >>>> >>> via >> >> >>> >>>> >>> >>>>>>> >>> messaging, once it is scalable (it will be v= ery >> >> very >> >> >>> >>>> >>> scalable!). So the >> >> >>> >>>> >>> >>>>>>> >>> GraphJobRunner can be partitioning the stuff >> with >> >> a >> >> >>> single >> >> >>> >>>> >>> superstep in >> >> >>> >>>> >>> >>>>>>> >>> setup() as it was before ages ago. The >> messaging >> >> must >> >> >>> be >> >> >>> >>>> >>> sorted anyway >> >> >>> >>>> >>> >>>>>>> for >> >> >>> >>>> >>> >>>>>>> >>> the algorithm so this is a nice side effect = and >> >> saves >> >> >>> us >> >> >>> >>>> the >> >> >>> >>>> >>> >>>>>>> partitioning >> >> >>> >>>> >>> >>>>>>> >>> job for graph processing. >> >> >>> >>>> >>> >>>>>>> >>> >> >> >>> >>>> >>> >>>>>>> >>> For other partitionings and with regard to o= ur >> >> >>> superstep >> >> >>> >>>> API, >> >> >>> >>>> >>> Suraj's >> >> >>> >>>> >>> >>>>>>> idea >> >> >>> >>>> >>> >>>>>>> >>> of injecting a preprocessing superstep that >> >> >>> partitions the >> >> >>> >>>> >>> stuff into >> >> >>> >>>> >>> >>>>>>> our >> >> >>> >>>> >>> >>>>>>> >>> messaging system is actually the best. >> >> >>> >>>> >>> >>>>>>> >>> >> >> >>> >>>> >>> >>>>>>> >>> >> >> >>> >>>> >>> >>>>>>> >>> 2013/3/6 Suraj Menon >> >> >>> >>>> >>> >>>>>>> >>> >> >> >>> >>>> >>> >>>>>>> >>>> No, the partitions we write locally need no= t >> be >> >> >>> sorted. >> >> >>> >>>> Sorry >> >> >>> >>>> >>> for the >> >> >>> >>>> >>> >>>>>>> >>>> confusion. The Superstep injection is possi= ble >> >> with >> >> >>> >>>> Superstep >> >> >>> >>>> >>> API. >> >> >>> >>>> >>> >>>>>>> There >> >> >>> >>>> >>> >>>>>>> >>>> are few enhancements needed to make it simp= ler >> >> after >> >> >>> I >> >> >>> >>>> last >> >> >>> >>>> >>> worked on >> >> >>> >>>> >>> >>>>>>> it. >> >> >>> >>>> >>> >>>>>>> >>>> We can then look into partitioning superste= p >> >> being >> >> >>> >>>> executed >> >> >>> >>>> >>> before the >> >> >>> >>>> >>> >>>>>>> >>>> setup of first superstep of submitted job. = I >> >> think >> >> >>> it is >> >> >>> >>>> >>> feasible. >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> On Tue, Mar 5, 2013 at 5:48 AM, Edward J. >> Yoon < >> >> >>> >>>> >>> edwardyoon@apache.org >> >> >>> >>>> >>> >>>>>>> >>>> >wrote: >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > > spilling queue and sorted spilling queu= e, >> >> can we >> >> >>> >>>> inject >> >> >>> >>>> >>> the >> >> >>> >>>> >>> >>>>>>> >>>> partitioning >> >> >>> >>>> >>> >>>>>>> >>>> > > superstep as the first superstep and us= e >> >> local >> >> >>> memory? >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > Actually, I wanted to add something befor= e >> >> calling >> >> >>> >>>> >>> BSP.setup() >> >> >>> >>>> >>> >>>>>>> method >> >> >>> >>>> >>> >>>>>>> >>>> > to avoid execute additional BSP job. But, >> in my >> >> >>> opinion, >> >> >>> >>>> >>> current is >> >> >>> >>>> >>> >>>>>>> >>>> > enough. I think, we need to collect more >> >> >>> experiences of >> >> >>> >>>> >>> input >> >> >>> >>>> >>> >>>>>>> >>>> > partitioning on large environments. I'll = do. >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > BTW, I still don't know why it need to be >> >> Sorted?! >> >> >>> >>>> MR-like? >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > On Thu, Feb 28, 2013 at 11:20 PM, Suraj >> Menon < >> >> >>> >>>> >>> >>>>>>> surajsmenon@apache.org> >> >> >>> >>>> >>> >>>>>>> >>>> > wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > > Sorry, I am increasing the scope here t= o >> >> outside >> >> >>> graph >> >> >>> >>>> >>> module. >> >> >>> >>>> >>> >>>>>>> When we >> >> >>> >>>> >>> >>>>>>> >>>> > have >> >> >>> >>>> >>> >>>>>>> >>>> > > spilling queue and sorted spilling queu= e, >> >> can we >> >> >>> >>>> inject >> >> >>> >>>> >>> the >> >> >>> >>>> >>> >>>>>>> >>>> partitioning >> >> >>> >>>> >>> >>>>>>> >>>> > > superstep as the first superstep and us= e >> >> local >> >> >>> memory? >> >> >>> >>>> >>> >>>>>>> >>>> > > Today we have partitioning job within a >> job >> >> and >> >> >>> are >> >> >>> >>>> >>> creating two >> >> >>> >>>> >>> >>>>>>> copies >> >> >>> >>>> >>> >>>>>>> >>>> > of >> >> >>> >>>> >>> >>>>>>> >>>> > > data on HDFS. This could be really cost= ly. >> >> Is it >> >> >>> >>>> possible >> >> >>> >>>> >>> to >> >> >>> >>>> >>> >>>>>>> create or >> >> >>> >>>> >>> >>>>>>> >>>> > > redistribute the partitions on local >> memory >> >> and >> >> >>> >>>> >>> initialize the >> >> >>> >>>> >>> >>>>>>> record >> >> >>> >>>> >>> >>>>>>> >>>> > > reader there? >> >> >>> >>>> >>> >>>>>>> >>>> > > The user can run a separate job give in >> >> examples >> >> >>> area >> >> >>> >>>> to >> >> >>> >>>> >>> >>>>>>> explicitly >> >> >>> >>>> >>> >>>>>>> >>>> > > repartition the data on HDFS. The >> deployment >> >> >>> question >> >> >>> >>>> is >> >> >>> >>>> >>> how much >> >> >>> >>>> >>> >>>>>>> of >> >> >>> >>>> >>> >>>>>>> >>>> disk >> >> >>> >>>> >>> >>>>>>> >>>> > > space gets allocated for local memory >> usage? >> >> >>> Would it >> >> >>> >>>> be >> >> >>> >>>> >>> a safe >> >> >>> >>>> >>> >>>>>>> >>>> approach >> >> >>> >>>> >>> >>>>>>> >>>> > > with the limitations? >> >> >>> >>>> >>> >>>>>>> >>>> > > >> >> >>> >>>> >>> >>>>>>> >>>> > > -Suraj >> >> >>> >>>> >>> >>>>>>> >>>> > > >> >> >>> >>>> >>> >>>>>>> >>>> > > On Thu, Feb 28, 2013 at 7:05 AM, Thomas >> >> Jungblut >> >> >>> >>>> >>> >>>>>>> >>>> > > wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > > >> >> >>> >>>> >>> >>>>>>> >>>> > >> yes. Once Suraj added merging of sorte= d >> >> files >> >> >>> we can >> >> >>> >>>> add >> >> >>> >>>> >>> this to >> >> >>> >>>> >>> >>>>>>> the >> >> >>> >>>> >>> >>>>>>> >>>> > >> partitioner pretty easily. >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> 2013/2/28 Edward J. Yoon < >> >> edwardyoon@apache.org >> >> >>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > Eh,..... btw, is re-partitioned data >> >> really >> >> >>> >>>> necessary >> >> >>> >>>> >>> to be >> >> >>> >>>> >>> >>>>>>> Sorted? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > On Thu, Feb 28, 2013 at 7:48 PM, Tho= mas >> >> >>> Jungblut >> >> >>> >>>> >>> >>>>>>> >>>> > >> > wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > Now I get how the partitioning wor= ks, >> >> >>> obviously >> >> >>> >>>> if >> >> >>> >>>> >>> you merge >> >> >>> >>>> >>> >>>>>>> n >> >> >>> >>>> >>> >>>>>>> >>>> > sorted >> >> >>> >>>> >>> >>>>>>> >>>> > >> > files >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > by just appending to each other, t= his >> >> will >> >> >>> >>>> result in >> >> >>> >>>> >>> totally >> >> >>> >>>> >>> >>>>>>> >>>> > unsorted >> >> >>> >>>> >>> >>>>>>> >>>> > >> > data >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > ;-) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > Why didn't you solve this via >> messaging? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > 2013/2/28 Thomas Jungblut < >> >> >>> >>>> thomas.jungblut@gmail.com >> >> >>> >>>> >>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> Seems that they are not correctly >> >> sorted: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 50 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 52 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 54 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 56 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 58 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 61 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> ... >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 78 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 81 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 83 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 85 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> ... >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 94 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 96 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 98 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 1 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 10 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 12 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 14 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 16 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 18 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 21 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 23 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 25 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 27 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 29 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> vertexID: 3 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> So this won't work then correctly= ... >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> 2013/2/28 Thomas Jungblut < >> >> >>> >>>> >>> thomas.jungblut@gmail.com> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> sure, have fun on your holidays. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> 2013/2/28 Edward J. Yoon < >> >> >>> >>>> edwardyoon@apache.org> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> Sure, but if you can fix quickl= y, >> >> please >> >> >>> do. >> >> >>> >>>> >>> March 1 is >> >> >>> >>>> >>> >>>>>>> >>>> > holiday[1] >> >> >>> >>>> >>> >>>>>>> >>>> > >> so >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> I'll appear next week. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> 1. >> >> >>> >>>> >>> >>>>>>> >> >> >>> http://en.wikipedia.org/wiki/Public_holidays_in_South_Korea >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> On Thu, Feb 28, 2013 at 6:36 PM= , >> >> Thomas >> >> >>> >>>> Jungblut >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> > Maybe 50 is missing from the >> file, >> >> >>> didn't >> >> >>> >>>> >>> observe if all >> >> >>> >>>> >>> >>>>>>> >>>> items >> >> >>> >>>> >>> >>>>>>> >>>> > >> were >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> added. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> > As far as I remember, I >> >> copy/pasted the >> >> >>> >>>> logic >> >> >>> >>>> >>> of the ID >> >> >>> >>>> >>> >>>>>>> into >> >> >>> >>>> >>> >>>>>>> >>>> > the >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> fastgen, >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> > want to have a look into it? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> > 2013/2/28 Edward J. Yoon < >> >> >>> >>>> edwardyoon@apache.org >> >> >>> >>>> >>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> I guess, it's a bug of fastg= en, >> >> when >> >> >>> >>>> generate >> >> >>> >>>> >>> adjacency >> >> >>> >>>> >>> >>>>>>> >>>> matrix >> >> >>> >>>> >>> >>>>>>> >>>> > >> into >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> multiple files. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> On Thu, Feb 28, 2013 at 6:29 >> PM, >> >> >>> Thomas >> >> >>> >>>> >>> Jungblut >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> > You have two files, are th= ey >> >> >>> partitioned >> >> >>> >>>> >>> correctly? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> > 2013/2/28 Edward J. Yoon < >> >> >>> >>>> >>> edwardyoon@apache.org> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> It looks like a bug. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax >> >> >>> :~/workspace/hama-trunk$ >> >> >>> >>>> ls >> >> >>> >>>> >>> -al >> >> >>> >>>> >>> >>>>>>> >>>> > >> /tmp/randomgraph/ >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> total 44 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edwa= rd >> >> 4096 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:03 . >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxrwt 19 root root >> >> 20480 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:04 .. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edwa= rd >> >> 2243 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:01 >> >> >>> >>>> >>> >>>>>>> part-00000 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edwa= rd >> >> 28 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:01 >> >> >>> >>>> >>> >>>>>>> >>>> > .part-00000.crc >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edwa= rd >> >> 2251 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:01 >> >> >>> >>>> >>> >>>>>>> part-00001 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edwa= rd >> >> 28 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:01 >> >> >>> >>>> >>> >>>>>>> >>>> > .part-00001.crc >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edwa= rd >> >> 4096 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:03 >> >> >>> >>>> >>> >>>>>>> partitions >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax >> >> >>> :~/workspace/hama-trunk$ >> >> >>> >>>> ls >> >> >>> >>>> >>> -al >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> /tmp/randomgraph/partitions/ >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> total 24 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 2 edward edwar= d >> 4096 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:03 . >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> drwxrwxr-x 3 edward edwar= d >> 4096 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> >>> 18:03 .. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edwar= d >> 2932 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> 18:03 >> >> >>> >>>> >>> >>>>>>> part-00000 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edwar= d >> 32 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> 18:03 >> >> >>> >>>> >>> >>>>>>> >>>> > .part-00000.crc >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rwxrwxrwx 1 edward edwar= d >> 2955 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> 18:03 >> >> >>> >>>> >>> >>>>>>> part-00001 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -rw-rw-r-- 1 edward edwar= d >> 32 >> >> >>> 2=EC=9B=94 28 >> >> >>> >>>> 18:03 >> >> >>> >>>> >>> >>>>>>> >>>> > .part-00001.crc >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> edward@udanax >> >> >>> :~/workspace/hama-trunk$ >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> On Thu, Feb 28, 2013 at 5= :27 >> >> PM, >> >> >>> Edward >> >> >>> >>>> < >> >> >>> >>>> >>> >>>>>>> >>>> edward@udanax.org >> >> >>> >>>> >>> >>>>>>> >>>> > > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> > yes i'll check again >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> > Sent from my iPhone >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> > On Feb 28, 2013, at 5:1= 8 >> PM, >> >> >>> Thomas >> >> >>> >>>> >>> Jungblut < >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> thomas.jungblut@gmail.com> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> wrote: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> Can you verify an >> >> observation >> >> >>> for me >> >> >>> >>>> >>> please? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> 2 files are created fr= om >> >> >>> fastgen, >> >> >>> >>>> >>> part-00000 and >> >> >>> >>>> >>> >>>>>>> >>>> > >> part-00001, >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> both >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> ~2.2kb >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> sized. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> In the below partition >> >> >>> directory, >> >> >>> >>>> there >> >> >>> >>>> >>> is only a >> >> >>> >>>> >>> >>>>>>> >>>> single >> >> >>> >>>> >>> >>>>>>> >>>> > >> > 5.56kb >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> file. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> Is it intended for the >> >> >>> partitioner to >> >> >>> >>>> >>> write a >> >> >>> >>>> >>> >>>>>>> single >> >> >>> >>>> >>> >>>>>>> >>>> > file >> >> >>> >>>> >>> >>>>>>> >>>> > >> if >> >> >>> >>>> >>> >>>>>>> >>>> > >> > you >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> configured >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> two? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> It even reads it as a = two >> >> files, >> >> >>> >>>> strange >> >> >>> >>>> >>> huh? >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> 2013/2/28 Thomas >> Jungblut < >> >> >>> >>>> >>> >>>>>>> thomas.jungblut@gmail.com> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> Will have a look into >> it. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> gen fastgen 100 10 >> >> >>> /tmp/randomgraph >> >> >>> >>>> 1 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> pagerank >> /tmp/randomgraph >> >> >>> >>>> /tmp/pageout >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> did work for me the l= ast >> >> time I >> >> >>> >>>> >>> profiled, maybe >> >> >>> >>>> >>> >>>>>>> the >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> partitioning >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> doesn't >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> partition correctly w= ith >> >> the >> >> >>> input >> >> >>> >>>> or >> >> >>> >>>> >>> something >> >> >>> >>>> >>> >>>>>>> else. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> 2013/2/28 Edward J. >> Yoon < >> >> >>> >>>> >>> edwardyoon@apache.org >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> Fastgen input seems n= ot >> >> work >> >> >>> for >> >> >>> >>>> graph >> >> >>> >>>> >>> examples. >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> edward@edward-virtualBox >> >> >>> >>>> >>> >>>>>>> :~/workspace/hama-trunk$ >> >> >>> >>>> >>> >>>>>>> >>>> > >> bin/hama >> >> >>> >>>> >>> >>>>>>> >>>> > >> > jar >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >> gen >> >> >>> >>>> >>> >>>>>>> >>>> > >> > fastgen >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> 100 10 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:02 WA= RN >> >> >>> >>>> >>> util.NativeCodeLoader: >> >> >>> >>>> >>> >>>>>>> Unable >> >> >>> >>>> >>> >>>>>>> >>>> > to >> >> >>> >>>> >>> >>>>>>> >>>> > >> > load >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop librar= y >> for >> >> your >> >> >>> >>>> >>> platform... >> >> >>> >>>> >>> >>>>>>> using >> >> >>> >>>> >>> >>>>>>> >>>> > >> > builtin-java >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> classes >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Running >> >> >>> >>>> >>> >>>>>>> >>>> job: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:03 IN= FO >> >> >>> >>>> >>> bsp.LocalBSPRunner: >> >> >>> >>>> >>> >>>>>>> Setting >> >> >>> >>>> >>> >>>>>>> >>>> up >> >> >>> >>>> >>> >>>>>>> >>>> > a >> >> >>> >>>> >>> >>>>>>> >>>> > >> new >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> barrier >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> for 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Current >> >> >>> >>>> >>> >>>>>>> >>>> > >> supersteps >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> number: 0 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: The >> >> >>> >>>> >>> >>>>>>> total >> >> >>> >>>> >>> >>>>>>> >>>> > number >> >> >>> >>>> >>> >>>>>>> >>>> > >> > of >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 0 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Counters: 3 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> org.apache.hama.bsp.JobInProgress$JobCounter >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > SUPERSTEPS=3D0 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=3D2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:06 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> TASK_OUTPUT_RECORDS=3D100 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Job Finished in 3.21= 2 >> >> seconds >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> edward@edward-virtualBox >> >> >>> >>>> >>> >>>>>>> :~/workspace/hama-trunk$ >> >> >>> >>>> >>> >>>>>>> >>>> > >> bin/hama >> >> >>> >>>> >>> >>>>>>> >>>> > >> > jar >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> examples/target/hama-examples-0.7.0-SNAPSHOT >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> hama-examples-0.7.0-SNAPSHOT-javadoc.jar >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> hama-examples-0.7.0-SNAPSHOT.jar >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> edward@edward-virtualBox >> >> >>> >>>> >>> >>>>>>> :~/workspace/hama-trunk$ >> >> >>> >>>> >>> >>>>>>> >>>> > >> bin/hama >> >> >>> >>>> >>> >>>>>>> >>>> > >> > jar >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> examples/target/hama-examples-0.7.0-SNAPSHOT.jar >> >> >>> >>>> >>> >>>>>>> >>>> > pagerank >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> /tmp/randomgraph >> >> /tmp/pageour >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 WA= RN >> >> >>> >>>> >>> util.NativeCodeLoader: >> >> >>> >>>> >>> >>>>>>> Unable >> >> >>> >>>> >>> >>>>>>> >>>> > to >> >> >>> >>>> >>> >>>>>>> >>>> > >> > load >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> native-hadoop librar= y >> for >> >> your >> >> >>> >>>> >>> platform... >> >> >>> >>>> >>> >>>>>>> using >> >> >>> >>>> >>> >>>>>>> >>>> > >> > builtin-java >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> classes >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> where applicable >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 IN= FO >> >> >>> >>>> >>> bsp.FileInputFormat: >> >> >>> >>>> >>> >>>>>>> Total >> >> >>> >>>> >>> >>>>>>> >>>> > input >> >> >>> >>>> >>> >>>>>>> >>>> > >> > paths >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> to >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> process >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:29 IN= FO >> >> >>> >>>> >>> bsp.FileInputFormat: >> >> >>> >>>> >>> >>>>>>> Total >> >> >>> >>>> >>> >>>>>>> >>>> > input >> >> >>> >>>> >>> >>>>>>> >>>> > >> > paths >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> to >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> process >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Running >> >> >>> >>>> >>> >>>>>>> >>>> job: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:30 IN= FO >> >> >>> >>>> >>> bsp.LocalBSPRunner: >> >> >>> >>>> >>> >>>>>>> Setting >> >> >>> >>>> >>> >>>>>>> >>>> up >> >> >>> >>>> >>> >>>>>>> >>>> > a >> >> >>> >>>> >>> >>>>>>> >>>> > >> new >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> barrier >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> for 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Current >> >> >>> >>>> >>> >>>>>>> >>>> > >> supersteps >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> number: 1 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: The >> >> >>> >>>> >>> >>>>>>> total >> >> >>> >>>> >>> >>>>>>> >>>> > number >> >> >>> >>>> >>> >>>>>>> >>>> > >> > of >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> supersteps: 1 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Counters: 6 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> org.apache.hama.bsp.JobInProgress$JobCounter >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > SUPERSTEPS=3D1 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > LAUNCHED_TASKS=3D2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> org.apache.hama.bsp.BSPPeerImpl$PeerCounter >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > SUPERSTEP_SUM=3D4 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> IO_BYTES_READ=3D4332 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> TIME_IN_SYNC_MS=3D14 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> TASK_INPUT_RECORDS=3D100 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.FileInputFormat: >> >> >>> >>>> >>> >>>>>>> Total >> >> >>> >>>> >>> >>>>>>> >>>> > input >> >> >>> >>>> >>> >>>>>>> >>>> > >> > paths >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> to >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> process >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> : 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.BSPJobClient: >> >> >>> >>>> >>> >>>>>>> Running >> >> >>> >>>> >>> >>>>>>> >>>> job: >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> job_localrunner_0001 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> bsp.LocalBSPRunner: >> >> >>> >>>> >>> >>>>>>> Setting >> >> >>> >>>> >>> >>>>>>> >>>> up >> >> >>> >>>> >>> >>>>>>> >>>> > a >> >> >>> >>>> >>> >>>>>>> >>>> > >> new >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> barrier >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> for 2 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> tasks! >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> graph.GraphJobRunner: 50 >> >> >>> >>>> >>> >>>>>>> >>>> > vertices >> >> >>> >>>> >>> >>>>>>> >>>> > >> > are >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> loaded >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> into >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:1 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 IN= FO >> >> >>> >>>> >>> graph.GraphJobRunner: 50 >> >> >>> >>>> >>> >>>>>>> >>>> > vertices >> >> >>> >>>> >>> >>>>>>> >>>> > >> > are >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> loaded >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> into >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> local:0 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> 13/02/28 10:32:33 ER= ROR >> >> >>> >>>> >>> bsp.LocalBSPRunner: >> >> >>> >>>> >>> >>>>>>> >>>> Exception >> >> >>> >>>> >>> >>>>>>> >>>> > >> > during >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> BSP >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> execution! >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> java.lang.IllegalArgumentException: >> >> >>> >>>> >>> Messages >> >> >>> >>>> >>> >>>>>>> must >> >> >>> >>>> >>> >>>>>>> >>>> > never >> >> >>> >>>> >>> >>>>>>> >>>> > >> be >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> behind >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> the >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> vertex in ID! Curren= t >> >> Message >> >> >>> ID: 1 >> >> >>> >>>> >>> vs. 50 >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >> >> >>> >> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:279) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >>>> >> >> >>> >> >> >> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:225= ) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >> >> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:129) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >>>> >> >> >>> >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256= ) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >>>> >> >> >>> >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:28= 6) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >>>> >> >> >>> >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:21= 1) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >>>> >> >> >>> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1110) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >> >> >>> >>>> >> >> >>> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:603) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> at >> >> >>> >>>> >>> java.lang.Thread.run(Thread.java:722) >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> -- >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> Best Regards, Edward= J. >> >> Yoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>>> @eddieyoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> -- >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> Best Regards, Edward J. Y= oon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> @eddieyoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> -- >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> @eddieyoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> -- >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> @eddieyoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >>> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> > -- >> >> >>> >>>> >>> >>>>>>> >>>> > >> > Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > @eddieyoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> > -- >> >> >>> >>>> >>> >>>>>>> >>>> > Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> >>>> > @eddieyoon >> >> >>> >>>> >>> >>>>>>> >>>> > >> >> >>> >>>> >>> >>>>>>> >>>> >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >> >> >> >>> >>>> >>> >>>>>>> >> -- >> >> >>> >>>> >>> >>>>>>> >> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> >> @eddieyoon >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> > >> >> >>> >>>> >>> >>>>>>> > -- >> >> >>> >>>> >>> >>>>>>> > Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> > @eddieyoon >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >>>>>>> -- >> >> >>> >>>> >>> >>>>>>> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>>>> @eddieyoon >> >> >>> >>>> >>> >>>>>>> >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> >> >> >>> >>>> >>> >>>>> -- >> >> >>> >>>> >>> >>>>> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>>> @eddieyoon >> >> >>> >>>> >>> >>>> >> >> >>> >>>> >>> >>>> >> >> >>> >>>> >>> >>>> >> >> >>> >>>> >>> >>>> -- >> >> >>> >>>> >>> >>>> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>>> @eddieyoon >> >> >>> >>>> >>> >>> >> >> >>> >>>> >>> >>> >> >> >>> >>>> >>> >>> >> >> >>> >>>> >>> >>> -- >> >> >>> >>>> >>> >>> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >>> @eddieyoon >> >> >>> >>>> >>> >> >> >> >>> >>>> >>> >> >> >> >>> >>>> >>> >> >> >> >>> >>>> >>> >> -- >> >> >>> >>>> >>> >> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> >> @eddieyoon >> >> >>> >>>> >>> > >> >> >>> >>>> >>> > >> >> >>> >>>> >>> > >> >> >>> >>>> >>> > -- >> >> >>> >>>> >>> > Best Regards, Edward J. Yoon >> >> >>> >>>> >>> > @eddieyoon >> >> >>> >>>> >>> >> >> >>> >>>> >>> >> >> >>> >>>> >>> >> >> >>> >>>> >>> -- >> >> >>> >>>> >>> Best Regards, Edward J. Yoon >> >> >>> >>>> >>> @eddieyoon >> >> >>> >>>> >>> >> >> >>> >>>> > >> >> >>> >>>> > >> >> >>> >>>> > >> >> >>> >>>> > -- >> >> >>> >>>> > Best Regards, Edward J. Yoon >> >> >>> >>>> > @eddieyoon >> >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> -- >> >> >>> >>>> Best Regards, Edward J. Yoon >> >> >>> >>>> @eddieyoon >> >> >>> >>>> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> -- >> >> >>> >> Best Regards, Edward J. Yoon >> >> >>> >> @eddieyoon >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> > -- >> >> >>> > Best Regards, Edward J. Yoon >> >> >>> > @eddieyoon >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> Best Regards, Edward J. Yoon >> >> >>> @eddieyoon >> >> >>> >> >> > >> >> > >> >> > >> >> > -- >> >> > Best Regards, Edward J. Yoon >> >> > @eddieyoon >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> @eddieyoon >> >> >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> --=20 Best Regards, Edward J. Yoon @eddieyoon