hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@gmail.com>
Subject Re: Hama receive queue
Date Mon, 16 Jul 2012 06:53:03 GMT
Hey,

I thought that you've dropped the project. Nice to see the progress, I will
have a deeper look when I'm back from work.
We checked a few math helpers into the new ML module last week, vectors and
matrices as well as their writables. Maybe you want to update your Hama to
the trunk version and code against the math library (if you think it would
be worth it).

So mainly the optimization step is left, cool stuff. Give you feedback
later then!
Thank you :)

2012/7/16 Aditya Sarawgi <sarawgi.aditya@gmail.com>

> Hi Thomas,
>
> As you suggested I have made some chnages to
>
> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/Icf.java
>
> The custom paritioner that you had for matrix multiplication seems to meet
> my needs as
> well. I just want to split each row such that each subsequent row goes to a
> new task.
> Would you like to review those changes ?
>
> The only thing left now is to take the input and output from hdfs (which is
> trivial i think )
> and to think about ways to generate test matrices for this.
>
>
> On Sun, Jul 15, 2012 at 7:57 PM, Aditya Sarawgi <sarawgi.aditya@gmail.com
> >wrote:
>
> > Hi Edward,
> >
> > Sorry for the late reply I have been busy with my internship. There are
> > still some fixes
> > left before we test it in full distributed mode. I will try to get that
> > done in the coming
> > few days.
> >
> > Thanks
> > Aditya Sarawgi
> >
> >
> > On Thu, Jul 5, 2012 at 9:08 AM, Edward J. Yoon <edwardyoon@apache.org
> >wrote:
> >
> >> > Now I would like to test this on a big input and possibly in full
> >> > distributed mode, so I was wondering how do
> >> > people usually do these sort of benchmarking.
> >>
> >> My 1 thousand cores cluster is ready.
> >>
> >> Do you have any plan to contribute your patch to Hama?
> >>
> >> On Thu, May 17, 2012 at 4:39 PM, Aditya Sarawgi
> >> <sarawgi.aditya@gmail.com> wrote:
> >> > Thanks Thomas.
> >> > I am actually using tags for something else. So for now using
> >> instanceof is
> >> > just fine with me.
> >> >
> >> > I had a couple of more questions, regarding benchmarking stuff on
> hama.
> >> I
> >> > have a working implementation  of
> >> > Parallel row based icf that given a n x n matrix returns a decomposed
> n
> >> x p
> >> > matrix.
> >> >
> >> >
> >>
> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/Icf.java
> >> >
> >> > Now I would like to test this on a big input and possibly in full
> >> > distributed mode, so I was wondering how do
> >> > people usually do these sort of benchmarking.
> >> >
> >> > Specifically,
> >> > 1) Do they setup a cluster on AWS ?
> >> > 2) What is like the standard way to load matrices in different nodes
> >> with a
> >> > custom partitioning scheme
> >> > 3) Is there anything else that I should know
> >> >
> >> > On Thu, May 17, 2012 at 3:20 AM, Thomas Jungblut <
> >> > thomas.jungblut@googlemail.com> wrote:
> >> >
> >> >> Hi Aditya,
> >> >>
> >> >> that's where the concept of Message Tagging comes into play. You have
> >> tags
> >> >> in each message which are hardcoded as Strings.
> >> >> But as Edward told you can use GenericWritable or ObjectWritable
> >> instead,
> >> >> so they will tag your messages with the classnames and give you the
> >> correct
> >> >> class.
> >> >>
> >> >> Is there any way by which I can pop from the receive queue ?
> >> >>
> >> >>
> >> >> peer.getCurrentMessage() is popping from the received queue.
> >> >>
> >> >> 2012/5/17 Aditya Sarawgi <sarawgi.aditya@gmail.com>
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > But thats not the only problem, consider this case
> >> >> > that there are variable number of messages being sent, so I would
> >> have to
> >> >> > maintain
> >> >> > counts for each peer pointing to the last unread message.
> >> >> >
> >> >> > Is there any way by which I can pop from the receive queue ?
> >> >> >
> >> >> >
> >> >> > On Wed, May 16, 2012 at 10:23 PM, Suraj Menon <
> >> surajsmenon@apache.org
> >> >> > >wrote:
> >> >> >
> >> >> > > Hi,
> >> >> > >
> >> >> > > Please take a look at this snippet of code copied and modified
> from
> >> >> > > Mapper class to implement your scenario. -
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/ssmenon/hama/edit/master/hama-mapreduce/src/org/apache/hama/computemodel/mapreduce/Trials.java
> >> >> > > Between lines 233 to 245 I am able to send different type
of
> >> messages.
> >> >> > > With type checks and generics you shouldn't be encountering
> >> Classcast
> >> >> > > exception at receiving end too. I am yet to test the next
> >> superstep,
> >> >> > > shall update you with sample code for the next superstep
> mimicking
> >> >> > > your scenario for receiving.
> >> >> > >
> >> >> > > For elegance, we have an experimental Superstep#compute
> >> >> > > API(org.apache.hama.bsp.Superstep). I have encountered an
issue
> in
> >> job
> >> >> > > submission framework with this method in distributed mode;
fix
> for
> >> >> > > this would be pushed to trunk in next few hours. You can
still
> run
> >> it
> >> >> > > using  LocalBSPRunner for now.
> >> >> > >
> >> >> > > -Suraj
> >> >> > >
> >> >> > > On Wed, May 16, 2012 at 9:18 PM, Aditya Sarawgi
> >> >> > > <sarawgi.aditya@gmail.com> wrote:
> >> >> > > > Hi Edward,
> >> >> > > >
> >> >> > > > Yes that is what I did
> >> >> > > > I wrote an ArrayMessage class (doesn't use generics
for now but
> >> can
> >> >> be
> >> >> > > > converted easily)
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
> https://github.com/truncs/hello-world/blob/master/src/main/java/edu/sunysb/cs/ArrayMessage.java
> >> >> > > >
> >> >> > > > But the problem is that I am sending a IntegerMessage
before
> and
> >> >> after
> >> >> > > > reading the IntegerMessage I am sending
> >> >> > > > an ArrayMessage but the previous IntegerMessage is still
there.
> >> >> > > >
> >> >> > > > On Wed, May 16, 2012 at 8:34 PM, Edward J. Yoon <
> >> >> edwardyoon@apache.org
> >> >> > > >wrote:
> >> >> > > >
> >> >> > > >> Hi,
> >> >> > > >>
> >> >> > > >> To send or receive multiple Message types, I think
you can use
> >> >> > > >> GenericWritable. You can also implement your own
> GenericMessage
> >> and
> >> >> > > >> contribute it to our project!
> >> >> > > >>
> >> >> > > >> Hope this helps you.
> >> >> > > >>
> >> >> > > >> On Thu, May 17, 2012 at 7:48 AM, Aditya Sarawgi
> >> >> > > >> <sarawgi.aditya@gmail.com> wrote:
> >> >> > > >> > Hi Guys,
> >> >> > > >> >
> >> >> > > >> > I am wondering how do the receive queues in
hama work.
> >> Consider
> >> >> this
> >> >> > > case
> >> >> > > >> > that I want to sent a different type of BSPMessage
in 2
> >> >> consecutive
> >> >> > > >> > superstep.
> >> >> > > >> > In this first superstep I am sending IntMessage
and in the
> >> next
> >> >> one
> >> >> > I
> >> >> > > am
> >> >> > > >> > sending a ArrayMessage ( custom message class).
> >> >> > > >> >
> >> >> > > >> > Now in the second super step when I do a
> >> >> > > >> >  while ((arrayMessage = (ArrayMessage)
> >> peer.getCurrentMessage())
> >> >> !=
> >> >> > > >> null) {
> >> >> > > >> >
> >> >> > > >> > it is throwing a java.lang.ClassCastException,
which is
> >> obvious
> >> >> > since
> >> >> > > its
> >> >> > > >> > trying to cast IntMessage to ArrayMessage.
> >> >> > > >> > I thought the message is dropped from the queue
after it is
> >> read,
> >> >> is
> >> >> > > this
> >> >> > > >> > not the case ?
> >> >> > > >> > And if it is not, how can this be handled elegantly
?
> >> >> > > >> >
> >> >> > > >> > --
> >> >> > > >> > Cheers,
> >> >> > > >> > Aditya Sarawgi
> >> >> > > >>
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> --
> >> >> > > >> Best Regards, Edward J. Yoon
> >> >> > > >> @eddieyoon
> >> >> > > >>
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > Cheers,
> >> >> > > > Aditya Sarawgi
> >> >> > >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Cheers,
> >> >> > Aditya Sarawgi
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thomas Jungblut
> >> >> Berlin <thomas.jungblut@gmail.com>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Cheers,
> >> > Aditya Sarawgi
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Cheers,
> > Aditya Sarawgi
> >
>
>
>
> --
> Cheers,
> Aditya Sarawgi
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message