Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hama-dev@incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CA+HzNQn8dxTdWn_=QFsn2EXgGNFY7NPqLx9ebcZBvPJQ7ddtUA@mail.gmail.com>
References: 
 <CA+HzNQnUi9gm+=11CV6wYHf9F=-L0uaUuvmqnTN4Bc-nCAbVJA@mail.gmail.com>
	<CAGQgZQRB-RKgpNb6bU0iCP=LvrcDp48uYJifiCsH=z4_OPMCPA@mail.gmail.com>
	<CA+HzNQ=-oxDMN6zJq0BXH9+8cspvFD=nkk9UW4ANij2XCRLF2Q@mail.gmail.com>
	<CALis4o5gmUFoUHh=ZzOqAp3yYKDY=wxTt6RuXb-zF8xG3wee8g@mail.gmail.com>
	<CA+HzNQk=Bk8Gg6GjLDJCPD+7P47ypfXpbLMRchM4f16PFvLewg@mail.gmail.com>
	<CAJ-=ysnJiZyUHwGMtrd=9BAQZghynx6JdjGs42G3C1uyFOjzwg@mail.gmail.com>
	<CA+HzNQn8dxTdWn_=QFsn2EXgGNFY7NPqLx9ebcZBvPJQ7ddtUA@mail.gmail.com>
Date: Thu, 17 May 2012 17:20:22 +0900
Message-ID: 
 <CAGQgZQTCmK+YYPJca+HgWL_gP-Y79MVwCTQirrXy8A4UkPuwQA@mail.gmail.com>
Subject: Re: Hama receive queue
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: hama-dev@incubator.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

If it's not urgent I'll help benchmark.

On Thu, May 17, 2012 at 4:39 PM, Aditya Sarawgi
<sarawgi.aditya@gmail.com> wrote:
> Thanks Thomas.
> I am actually using tags for something else. So for now using instanceof =
is
> just fine with me.
>
> I had a couple of more questions, regarding benchmarking stuff on hama. I
> have a working implementation =C2=A0of
> Parallel row based icf that given a n x n matrix returns a decomposed n x=
 p
> matrix.
>
> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunys=
b/cs/Icf.java
>
> Now I would like to test this on a big input and possibly in full
> distributed mode, so I was wondering how do
> people usually do these sort of benchmarking.
>
> Specifically,
> 1) Do they setup a cluster on AWS ?
> 2) What is like the standard way to load matrices in different nodes with=
 a
> custom partitioning scheme
> 3) Is there anything else that I should know
>
> On Thu, May 17, 2012 at 3:20 AM, Thomas Jungblut <
> thomas.jungblut@googlemail.com> wrote:
>
>> Hi Aditya,
>>
>> that's where the concept of Message Tagging comes into play. You have ta=
gs
>> in each message which are hardcoded as Strings.
>> But as Edward told you can use GenericWritable or ObjectWritable instead=
,
>> so they will tag your messages with the classnames and give you the corr=
ect
>> class.
>>
>> Is there any way by which I can pop from the receive queue ?
>>
>>
>> peer.getCurrentMessage() is popping from the received queue.
>>
>> 2012/5/17 Aditya Sarawgi <sarawgi.aditya@gmail.com>
>>
>> > Hi,
>> >
>> > But thats not the only problem, consider this case
>> > that there are variable number of messages being sent, so I would have=
 to
>> > maintain
>> > counts for each peer pointing to the last unread message.
>> >
>> > Is there any way by which I can pop from the receive queue ?
>> >
>> >
>> > On Wed, May 16, 2012 at 10:23 PM, Suraj Menon <surajsmenon@apache.org
>> > >wrote:
>> >
>> > > Hi,
>> > >
>> > > Please take a look at this snippet of code copied and modified from
>> > > Mapper class to implement your scenario. -
>> > >
>> > >
>> >
>> https://github.com/ssmenon/hama/edit/master/hama-mapreduce/src/org/apach=
e/hama/computemodel/mapreduce/Trials.java
>> > > Between lines 233 to 245 I am able to send different type of message=
s.
>> > > With type checks and generics you shouldn't be encountering Classcas=
t
>> > > exception at receiving end too. I am yet to test the next superstep,
>> > > shall update you with sample code for the next superstep mimicking
>> > > your scenario for receiving.
>> > >
>> > > For elegance, we have an experimental Superstep#compute
>> > > API(org.apache.hama.bsp.Superstep). I have encountered an issue in j=
ob
>> > > submission framework with this method in distributed mode; fix for
>> > > this would be pushed to trunk in next few hours. You can still run i=
t
>> > > using =C2=A0LocalBSPRunner for now.
>> > >
>> > > -Suraj
>> > >
>> > > On Wed, May 16, 2012 at 9:18 PM, Aditya Sarawgi
>> > > <sarawgi.aditya@gmail.com> wrote:
>> > > > Hi Edward,
>> > > >
>> > > > Yes that is what I did
>> > > > I wrote an ArrayMessage class (doesn't use generics for now but ca=
n
>> be
>> > > > converted easily)
>> > > >
>> > >
>> >
>> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/suny=
sb/cs/ArrayMessage.java
>> > > >
>> > > > But the problem is that I am sending a IntegerMessage before and
>> after
>> > > > reading the IntegerMessage I am sending
>> > > > an ArrayMessage but the previous IntegerMessage is still there.
>> > > >
>> > > > On Wed, May 16, 2012 at 8:34 PM, Edward J. Yoon <
>> edwardyoon@apache.org
>> > > >wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> To send or receive multiple Message types, I think you can use
>> > > >> GenericWritable. You can also implement your own GenericMessage a=
nd
>> > > >> contribute it to our project!
>> > > >>
>> > > >> Hope this helps you.
>> > > >>
>> > > >> On Thu, May 17, 2012 at 7:48 AM, Aditya Sarawgi
>> > > >> <sarawgi.aditya@gmail.com> wrote:
>> > > >> > Hi Guys,
>> > > >> >
>> > > >> > I am wondering how do the receive queues in hama work. Consider
>> this
>> > > case
>> > > >> > that I want to sent a different type of BSPMessage in 2
>> consecutive
>> > > >> > superstep.
>> > > >> > In this first superstep I am sending IntMessage and in the next
>> one
>> > I
>> > > am
>> > > >> > sending a ArrayMessage ( custom message class).
>> > > >> >
>> > > >> > Now in the second super step when I do a
>> > > >> > =C2=A0while ((arrayMessage =3D (ArrayMessage) peer.getCurrentMe=
ssage())
>> !=3D
>> > > >> null) {
>> > > >> >
>> > > >> > it is throwing a java.lang.ClassCastException, which is obvious
>> > since
>> > > its
>> > > >> > trying to cast IntMessage to ArrayMessage.
>> > > >> > I thought the message is dropped from the queue after it is rea=
d,
>> is
>> > > this
>> > > >> > not the case ?
>> > > >> > And if it is not, how can this be handled elegantly ?
>> > > >> >
>> > > >> > --
>> > > >> > Cheers,
>> > > >> > Aditya Sarawgi
>> > > >>
>> > > >>
>> > > >>
>> > > >> --
>> > > >> Best Regards, Edward J. Yoon
>> > > >> @eddieyoon
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Cheers,
>> > > > Aditya Sarawgi
>> > >
>> >
>> >
>> >
>> > --
>> > Cheers,
>> > Aditya Sarawgi
>> >
>>
>>
>>
>> --
>> Thomas Jungblut
>> Berlin <thomas.jungblut@gmail.com>
>>
>
>
>
> --
> Cheers,
> Aditya Sarawgi


--=20
Best Regards, Edward J. Yoon
@eddieyoon