Mailing-List: contact user-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hama.apache.org
Received-SPF: pass (nike.apache.org: domain of thomas.jungblut@gmail.com
 designates 209.85.216.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <B9B4E2C0-0933-4303-A776-F80E37486D2B@disi.unitn.it>
References: <62685C8E-DFCE-43D2-AFEB-0288CCCB2CDB@disi.unitn.it>
	<CAJ-=ysmPzmqiEzUuG62PF8E4M=taxvR4_20rKDi2cu03bAPFXQ@mail.gmail.com>
	<DD2B79A5-6303-4FE7-BA49-D6D15CAAB7C3@disi.unitn.it>
	<CAJ-=ysk0CPYEOwCpHN61B+fzrrae_1=tOcyLKY1LjayGaUYMpw@mail.gmail.com>
	<939E5043-1B89-4ED1-B38A-755432CB34FC@disi.unitn.it>
	<CAJ-=yskFZenpCtnLy35er0gM3AG9dDzucDYxaEXQ2m2nU25Lxg@mail.gmail.com>
	<B9B4E2C0-0933-4303-A776-F80E37486D2B@disi.unitn.it>
Date: Wed, 5 Dec 2012 13:56:37 +0100
Message-ID: 
 <CAJ-=ysmxKwtuDvCwO1LNAMyV5XoTrW5un3ceY36FUdMD=By_4g@mail.gmail.com>
Subject: Re: Partitioning
From: Thomas Jungblut <thomas.jungblut@gmail.com>
To: user@hama.apache.org
Content-Type: multipart/alternative; boundary=485b397dd4df53a94204d01a833b

--485b397dd4df53a94204d01a833b
Content-Type: text/plain; charset=ISO-8859-1

Exactly,

maybe you want to first read up all the different modes and how they are
configured:

http://wiki.apache.org/hama/GettingStarted#Modes

We also have some nice documentations as PDF which you can get here:

http://wiki.apache.org/hama/GettingStarted#Hama_0.6.0

The configuration property to change the number of tasks on every host is
"bsp.tasks.maximum" which is described by ">The maximum number of BSP tasks
that will be run simultaneously by a groom server.".

Setting this to 1 on every host where a groom server starts, and afterwards
restarting your cluster should do what you want to archieve.
I can recommend puppet for maintaining these kinds of configurations.

If you need a more formal complexity model for BSP applications let me
know, I have derived one from Rob Bisseling's BSP model that fits better to
Apache Hama's style of computation.


2012/12/5 Benedikt Elser <elser@disi.unitn.it>

> Ah, local mode, Bingo!
>
> About the communication costs: Yes I am aware of these, however this is
> exactly what I want to test in the first place :) Hence I would need a
> bsp.distributed.tasks.maximum
>
> Thanks for the clarifications,
>
> Benedikt
>
> On Dec 5, 2012, at 12:05 PM, Thomas Jungblut wrote:
>
> > Because the property is called "local". This doesn't affect the
> distributed
> > mode.
> > Note that it is really bad if you compute multiple tasks on different
> host
> > machines, because this leverages your communication costs.
> >
> > 2012/12/5 Benedikt Elser <elser@disi.unitn.it>
> >
> >> Thank you, I will try that. However if I set bsp.local.tasks.maximum to
> 1,
> >> why doesn't it distribute one task to each machine?
> >>
> >> On Dec 5, 2012, at 11:58 AM, Thomas Jungblut wrote:
> >>
> >>> So it will spawn 12 tasks. If this doesn't satisfy the load on your
> >>> machines, try to use smaller blocksizes.
> >>>
> >>> 2012/12/5 Benedikt Elser <elser@disi.unitn.it>
> >>>
> >>>> Hi,
> >>>>
> >>>> thanks for your reply!
> >>>>
> >>>> Total size:    49078776 B
> >>>> Total dirs:    1
> >>>> Total files:   12
> >>>> Total blocks (validated):      12 (avg. block size 4089898 B)
> >>>>
> >>>> Benedikt
> >>>>
> >>>> On Dec 5, 2012, at 11:47 AM, Thomas Jungblut wrote:
> >>>>
> >>>>> So how many blocks has your data in HDFS?
> >>>>>
> >>>>> 2012/12/5 Benedikt Elser <elser@disi.unitn.it>
> >>>>>
> >>>>>> Hi List,
> >>>>>>
> >>>>>> I am using the hama-0.6.0 release to run graph jobs on various input
> >>>>>> graphs in a ec2 based cluster of size 12. However as I see in the
> logs
> >>>> not
> >>>>>> every node on the cluster contributes to that job (they have no
> >>>>>> tasklog/job<ID> dir and are idle). Theoretically a distribution of 1
> >>>>>> Million nodes across 12 buckets should hit every node at least once.
> >>>>>> Therefore I think its a configuration problem. So far I messed
> around
> >>>> with
> >>>>>> these settings:
> >>>>>>
> >>>>>> <name>bsp.max.tasks.per.job</name>
> >>>>>> <name>bsp.local.tasks.maximum</name>
> >>>>>> <name>bsp.tasks.maximum</name>
> >>>>>> <name>bsp.child.java.opts</name>
> >>>>>>
> >>>>>> Setting bsp.local.tasks.maximum to 1 and bsp.tasks.maximum.per.job
> to
> >> 12
> >>>>>> hat not the desired effect. I also split the input into 12 files
> >>>> (because
> >>>>>> of something in 0.5, that was fixed in 0.6).
> >>>>>>
> >>>>>> Could you recommend me some settings or guide me through the
> system's
> >>>>>> partition decision? I thought it would be:
> >>>>>>
> >>>>>> Input -> Input Split based on input, max* conf values -> number of
> >> tasks
> >>>>>> HashPartition.class distributes Ids across that number of tasks.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Benedikt
> >>>>
> >>>>
> >>
> >>
>
>

--485b397dd4df53a94204d01a833b--