incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@googlemail.com>
Subject Re: counters and supersteps
Date Wed, 07 Dec 2011 10:26:32 GMT
I updated this to a shared object because I wanted to make flow control
with the counters instead of sending messages to a master task.
But that isn't actually how the normal hama is working. so I'm going to
revert this.

Thanks though ;)

2011/12/7 Tommaso Teofili <tommaso.teofili@gmail.com>

> right, it looks like Edward's patch fixes it.
>
> ...
>
> 11/12/07 11:06:06 INFO examples.RandBench$RandBSP: local:18 to local:10 :
> 10
>
> 11/12/07 11:06:06 INFO examples.RandBench$RandBSP: local:18 to local:10 :
> 10
>
> 11/12/07 11:06:06 INFO examples.RandBench$RandBSP: local:18 to local:10 :
> 10
>
> 11/12/07 11:06:08 INFO bsp.BSPJobClient: Current supersteps number: 200
>
> 11/12/07 11:06:08 INFO bsp.BSPJobClient: The total number of supersteps:
> 200
>
> Job Finished in 6.38 seconds
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.56 sec
>
>
> Thanks,
> Tommaso
>
>
> 2011/12/7 Thomas Jungblut <thomas.jungblut@googlemail.com>
>
> > Ah sorry, but since this is a shared object, this will be incremented by
> > all tasks.
> > So your fix will fix it ;)
> > Sorry.
> >
> > 2011/12/7 Thomas Jungblut <thomas.jungblut@googlemail.com>
> >
> > > Wait a second please:
> > > The localrunner is going to piggyback to the client console with a
> field
> > > called "superStepCount".
> > > This is just getting updated when a new taskstatus comes in, and if it
> is
> > > greater than superStepCount it updates the variable.
> > > There is no summation.
> > >
> > > Are you speaking about the LocalBSPRunner Tommaso?
> > >
> > >
> > > 2011/12/7 Edward J. Yoon <edwardyoon@apache.org>
> > >
> > >> Counters seems synchronized. Below small patch will fix this problem.
> > >>
> > >> Index:
> > >>
> >
> src/main/java/org/apache/hama/bsp/LocalBSPRunner.java===================================================================---
> > >> src/main/java/org/apache/hama/bsp/LocalBSPRunner.java   (revision
> > >> 1211275)+++ src/main/java/org/apache/hama/bsp/LocalBSPRunner.java
> > >> (working
> > >> copy)@@ -213,7 +213,6 @@     private int id;     private BSP bsp;
> > >> private RawSplit[] splits;-    private static final Counters counters
> > >> = new Counters();      public BSPRunner(Configuration conf, BSPJob
> > >> job, int id, RawSplit[] splits) {       super();@@ -244,7 +243,7 @@
> > >>     BSPPeerImpl peer = new BSPPeerImpl(job, conf, new TaskAttemptID(
> > >>         new TaskID(job.getJobID(), id), id), new LocalUmbilical(),
> > >> id,-          splitname, realBytes, counters);+          splitname,
> > >> realBytes, new Counters());        bsp.setConf(conf);       try {
> > >>
> > >> On Wed, Dec 7, 2011 at 6:05 PM, Tommaso Teofili
> > >> <tommaso.teofili@gmail.com> wrote:
> > >> > for an example take a look at the RandBenchTest which runs RandBench
> > >> with
> > >> > 200 supersteps  and it ends up with :
> > >> >
> > >> > ...
> > >> > 11/12/07 09:59:07 INFO examples.RandBench$RandBSP: local:19 to
> local:7
> > >> : 10
> > >> >
> > >> > 11/12/07 09:59:07 INFO examples.RandBench$RandBSP: local:19 to
> local:7
> > >> : 10
> > >> >
> > >> > 11/12/07 09:59:07 INFO examples.RandBench$RandBSP: local:19 to
> local:7
> > >> : 10
> > >> >
> > >> > 11/12/07 09:59:08 INFO bsp.BSPJobClient: Current supersteps number:
> > 4000
> > >> >
> > >> > 11/12/07 09:59:08 INFO bsp.BSPJobClient: The total number of
> > supersteps:
> > >> > 4000
> > >> >
> > >> >
> > >> > which is exactly 200*cluster.getMaxTasks() on my laptop.
> > >> >
> > >> > Tommaso
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > 2011/12/7 Thomas Jungblut <thomas.jungblut@googlemail.com>
> > >> >
> > >> >> We are so buggy :D
> > >> >>
> > >> >> 2011/12/7 Edward J. Yoon <edwardyoon@apache.org>
> > >> >>
> > >> >> > OMG!
> > >> >> >
> > >> >> > On Wed, Dec 7, 2011 at 5:31 PM, Tommaso Teofili
> > >> >> > <tommaso.teofili@gmail.com> wrote:
> > >> >> > > Hi all,
> > >> >> > >
> > >> >> > > after the recent updates in trunk it seems to me the
count of
> the
> > >> >> number
> > >> >> > of
> > >> >> > > supersteps is not working correctly anymore.
> > >> >> > > I have a sample algorithm with 3 supersteps split to
200 tasks
> > and
> > >> I
> > >> >> got
> > >> >> > a
> > >> >> > > count of 600 supersteps in the logger.
> > >> >> > >
> > >> >> > > Anyone experiencing the same issue?
> > >> >> > > Tommaso
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > Best Regards, Edward J. Yoon
> > >> >> > @eddieyoon
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Thomas Jungblut
> > >> >> Berlin <thomas.jungblut@gmail.com>
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards, Edward J. Yoon
> > >> @eddieyoon
> > >>
> > >
> > >
> > >
> > > --
> > > Thomas Jungblut
> > > Berlin <thomas.jungblut@gmail.com>
> > >
> >
> >
> >
> > --
> > Thomas Jungblut
> > Berlin <thomas.jungblut@gmail.com>
> >
>



-- 
Thomas Jungblut
Berlin <thomas.jungblut@gmail.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message