Mailing-List: contact dev-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hama.apache.org
Received-SPF: pass (athena.apache.org: domain of thomas.jungblut@gmail.com
 designates 209.85.216.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAGQgZQSEVvk1avfWkdAVDn6n-2ewMGrqOUtB1hCaVSQWHBXVjA@mail.gmail.com>
References: 
 <CAGQgZQR7sZuOR04VS5_GtCY3F6tXHY=CJQruOExyzqSaChfVDA@mail.gmail.com>
	<CAGQgZQQeYsJKbEx9ke4TWGCBg5VrZv5iUqp42M9BbE-7f3epYQ@mail.gmail.com>
	<CAGQgZQRD_YAuRiaUmvWJym1O9eOwJek6vL68v=JmFNTF1G3odg@mail.gmail.com>
	<CAJ-=ysmKWW4MUF7jwtx_ayz=dpgOfgaASchu0=ymayRdmbH4ZQ@mail.gmail.com>
	<CAGQgZQTRq7WeZzzCj-vCgKMAD5Nx1U3C07yjqXD3M1XuW79oHQ@mail.gmail.com>
	<CAJ-=ys=0s2WBdKjh6aZfx+MEUxbQbohhnfc886XL85tamkan6A@mail.gmail.com>
	<CAGQgZQSEVvk1avfWkdAVDn6n-2ewMGrqOUtB1hCaVSQWHBXVjA@mail.gmail.com>
Date: Mon, 10 Dec 2012 12:21:55 +0100
Message-ID: 
 <CAJ-=ysnt0P5EX5KWNv2=KyF_n4+3z1TwJxB6LEPHt9VLUgy+ow@mail.gmail.com>
Subject: Re: runtimePartitioning in GraphJobRunner
From: Thomas Jungblut <thomas.jungblut@gmail.com>
To: dev@hama.apache.org
Content-Type: multipart/alternative; boundary=0016369f9b54dcc82c04d07dc5fa

--0016369f9b54dcc82c04d07dc5fa
Content-Type: text/plain; charset=ISO-8859-1

Yes, because changing the blocksize to 32m will just use 300mb of memory,
so you can add more machines to fit the number of resulting tasks.

If each node have small memory, there's no way to process in memory


Yes, so spilling on disk is the easiest solution to save memory. Not
changing the partitioning.
If you want to split again through the block boundaries to distribute the
data through the cluster, then do it, but this is plainly wrong.

2012/12/10 Edward J. Yoon <edwardyoon@apache.org>

> > A Hama cluster is scalable. It means that the computing capacity
> >> should be increased by adding slaves. Right?
> >
> >
> > I'm sorry, but I don't see how this relates to the vertex input reader.
>
> Not related with input reader. It related with partitioning and load
> balancing. As I reported to you before, to process vertices within
> 256MB block, each TaskRunner requied 25~30GB memory.
>
> If each node have small memory, there's no way to process in memory
> without changing block size of HDFS.
>
> Do you think this is scalable?
>
> On Mon, Dec 10, 2012 at 7:59 PM, Thomas Jungblut
> <thomas.jungblut@gmail.com> wrote:
> > Oh okay, so if you want to remove that, have a lot of fun. This reader is
> > needed, so people can create vertices from their own fileformat.
> > Going back to a sequencefile input will not only break backward
> > compatibility but also make the same issues we had before.
> >
> > A Hama cluster is scalable. It means that the computing capacity
> >> should be increased by adding slaves. Right?
> >
> >
> > I'm sorry, but I don't see how this relates to the vertex input reader.
> >
> > 2012/12/10 Edward J. Yoon <edwardyoon@apache.org>
> >
> >> A Hama cluster is scalable. It means that the computing capacity
> >> should be increased by adding slaves. Right?
> >>
> >> As I mentioned before, disk-queue and storing vertices on local disk
> >> are not urgent.
> >>
> >> In short, yeah, I wan to remove VertexInputReader and runtime
> >> partition in Graph package.
> >>
> >> See also,
> >>
> https://issues.apache.org/jira/browse/HAMA-531?focusedCommentId=13527756&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13527756
> >>
> >> On Mon, Dec 10, 2012 at 7:31 PM, Thomas Jungblut
> >> <thomas.jungblut@gmail.com> wrote:
> >> > uhm, I have no idea what you want to archieve, do you want to get
> back to
> >> > client-side partitioning?
> >> >
> >> > 2012/12/10 Edward J. Yoon <edwardyoon@apache.org>
> >> >
> >> >> If there's no opinion, I'll remove VertexInputReader in
> >> >> GraphJobRunner, because it make code complex. Let's consider again
> >> >> about the VertexInputReader, after fixing HAMA-531 and HAMA-632
> >> >> issues.
> >> >>
> >> >> On Fri, Dec 7, 2012 at 9:35 AM, Edward J. Yoon <
> edwardyoon@apache.org>
> >> >> wrote:
> >> >> > Or, I'd like to get rid of VertexInputReader.
> >> >> >
> >> >> > On Fri, Dec 7, 2012 at 9:30 AM, Edward J. Yoon <
> edwardyoon@apache.org
> >> >
> >> >> wrote:
> >> >> >> In fact, there's no choice but to use runtimePartitioning
> (because of
> >> >> >> VertexInputReader). Right? If so, I would like to delete all "if
> >> >> >> (runtimePartitioning) {" conditions.
> >> >> >>
> >> >> >> --
> >> >> >> Best Regards, Edward J. Yoon
> >> >> >> @eddieyoon
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best Regards, Edward J. Yoon
> >> >> > @eddieyoon
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best Regards, Edward J. Yoon
> >> >> @eddieyoon
> >> >>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

--0016369f9b54dcc82c04d07dc5fa--