hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya Sarawgi <sarawgi.adi...@gmail.com>
Subject Re: Hama receive queue
Date Thu, 17 May 2012 07:39:45 GMT
Thanks Thomas.
I am actually using tags for something else. So for now using instanceof is
just fine with me.

I had a couple of more questions, regarding benchmarking stuff on hama. I
have a working implementation  of
Parallel row based icf that given a n x n matrix returns a decomposed n x p
matrix.

https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/Icf.java

Now I would like to test this on a big input and possibly in full
distributed mode, so I was wondering how do
people usually do these sort of benchmarking.

Specifically,
1) Do they setup a cluster on AWS ?
2) What is like the standard way to load matrices in different nodes with a
custom partitioning scheme
3) Is there anything else that I should know

On Thu, May 17, 2012 at 3:20 AM, Thomas Jungblut <
thomas.jungblut@googlemail.com> wrote:

> Hi Aditya,
>
> that's where the concept of Message Tagging comes into play. You have tags
> in each message which are hardcoded as Strings.
> But as Edward told you can use GenericWritable or ObjectWritable instead,
> so they will tag your messages with the classnames and give you the correct
> class.
>
> Is there any way by which I can pop from the receive queue ?
>
>
> peer.getCurrentMessage() is popping from the received queue.
>
> 2012/5/17 Aditya Sarawgi <sarawgi.aditya@gmail.com>
>
> > Hi,
> >
> > But thats not the only problem, consider this case
> > that there are variable number of messages being sent, so I would have to
> > maintain
> > counts for each peer pointing to the last unread message.
> >
> > Is there any way by which I can pop from the receive queue ?
> >
> >
> > On Wed, May 16, 2012 at 10:23 PM, Suraj Menon <surajsmenon@apache.org
> > >wrote:
> >
> > > Hi,
> > >
> > > Please take a look at this snippet of code copied and modified from
> > > Mapper class to implement your scenario. -
> > >
> > >
> >
> https://github.com/ssmenon/hama/edit/master/hama-mapreduce/src/org/apache/hama/computemodel/mapreduce/Trials.java
> > > Between lines 233 to 245 I am able to send different type of messages.
> > > With type checks and generics you shouldn't be encountering Classcast
> > > exception at receiving end too. I am yet to test the next superstep,
> > > shall update you with sample code for the next superstep mimicking
> > > your scenario for receiving.
> > >
> > > For elegance, we have an experimental Superstep#compute
> > > API(org.apache.hama.bsp.Superstep). I have encountered an issue in job
> > > submission framework with this method in distributed mode; fix for
> > > this would be pushed to trunk in next few hours. You can still run it
> > > using  LocalBSPRunner for now.
> > >
> > > -Suraj
> > >
> > > On Wed, May 16, 2012 at 9:18 PM, Aditya Sarawgi
> > > <sarawgi.aditya@gmail.com> wrote:
> > > > Hi Edward,
> > > >
> > > > Yes that is what I did
> > > > I wrote an ArrayMessage class (doesn't use generics for now but can
> be
> > > > converted easily)
> > > >
> > >
> >
> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/ArrayMessage.java
> > > >
> > > > But the problem is that I am sending a IntegerMessage before and
> after
> > > > reading the IntegerMessage I am sending
> > > > an ArrayMessage but the previous IntegerMessage is still there.
> > > >
> > > > On Wed, May 16, 2012 at 8:34 PM, Edward J. Yoon <
> edwardyoon@apache.org
> > > >wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> To send or receive multiple Message types, I think you can use
> > > >> GenericWritable. You can also implement your own GenericMessage and
> > > >> contribute it to our project!
> > > >>
> > > >> Hope this helps you.
> > > >>
> > > >> On Thu, May 17, 2012 at 7:48 AM, Aditya Sarawgi
> > > >> <sarawgi.aditya@gmail.com> wrote:
> > > >> > Hi Guys,
> > > >> >
> > > >> > I am wondering how do the receive queues in hama work. Consider
> this
> > > case
> > > >> > that I want to sent a different type of BSPMessage in 2
> consecutive
> > > >> > superstep.
> > > >> > In this first superstep I am sending IntMessage and in the next
> one
> > I
> > > am
> > > >> > sending a ArrayMessage ( custom message class).
> > > >> >
> > > >> > Now in the second super step when I do a
> > > >> >  while ((arrayMessage = (ArrayMessage) peer.getCurrentMessage())
> !=
> > > >> null) {
> > > >> >
> > > >> > it is throwing a java.lang.ClassCastException, which is obvious
> > since
> > > its
> > > >> > trying to cast IntMessage to ArrayMessage.
> > > >> > I thought the message is dropped from the queue after it is read,
> is
> > > this
> > > >> > not the case ?
> > > >> > And if it is not, how can this be handled elegantly ?
> > > >> >
> > > >> > --
> > > >> > Cheers,
> > > >> > Aditya Sarawgi
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best Regards, Edward J. Yoon
> > > >> @eddieyoon
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers,
> > > > Aditya Sarawgi
> > >
> >
> >
> >
> > --
> > Cheers,
> > Aditya Sarawgi
> >
>
>
>
> --
> Thomas Jungblut
> Berlin <thomas.jungblut@gmail.com>
>



-- 
Cheers,
Aditya Sarawgi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message