Mailing-List: contact dev-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hama.apache.org
Received-SPF: pass (athena.apache.org: domain of tommaso.teofili@gmail.com
 designates 209.85.220.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+jdVczAJNqyzDUgCuMneKU0OnnHSiKfioxbkr0zEz-=k8PXqg@mail.gmail.com>
References: <BLU0-SMTP419DF744B2F90DF2EA9582BE9650@phx.gbl>
 <CA+jdVczAJNqyzDUgCuMneKU0OnnHSiKfioxbkr0zEz-=k8PXqg@mail.gmail.com>
From: Tommaso Teofili <tommaso.teofili@gmail.com>
Date: Mon, 15 Jul 2013 09:51:59 +0200
Message-ID: 
 <CAGnSx05XZECMqjiz6CE4A8GJcbpoXGY1p5=i-=o7i9BCBzFugw@mail.gmail.com>
Subject: Re: Dynamic vertices and hama counters
To: dev@hama.apache.org
Content-Type: multipart/alternative; boundary=047d7b6da8c203f59104e188251a

--047d7b6da8c203f59104e188251a
Content-Type: text/plain; charset=ISO-8859-1

what about introducing a proper API for counting vertices, something like
an interface VertexCounter with 2-3 implementations like
InMemoryVertexCounter (basically the current one), a
DistributedVertexCounter to implement the scenario where we use a separate
BSP superstep to count them and a ZKVertexCounter which handles vertices
counts as per Chian-Hung's suggestion.

Also we may introduce something like a configuration variable to define if
all the vertices are needed or just the neighbors (and/or some other
strategy).

My 2 cents,
Tommaso

2013/7/14 Chia-Hung Lin <clin4j@googlemail.com>

> Just my personal viewpoint. For small size of global information,
> considering to store the state in ZooKeeper might be a reasonable
> solution.
>
> On 13 July 2013 21:28, andronat_asf <andronat_asf@hotmail.com> wrote:
> > Hello everyone,
> >
> > I'm working on HAMA-767 and I have some concerns on counters and
> scalability. Currently, every peer has a set of vertices and a variable
> that is keeping the total number of vertices through all peers. In my case,
> I'm trying to add and remove vertices during the runtime of a job, which
> means that I have to update all those variables.
> >
> > My problem is that this is not efficient because in every operation (add
> or remove a vertex) I need to update all peers, so I need to send lots of
> messages to make those updates (see GraphJobRunner#countGlobalVertexCount
> method) and I believe this is not correct and scalable. An other problem is
> that, even if I update all those variable (with the cost of sending lots of
> messages to every peer) those variables will be updated on the next
> superstep.
> >
> > e.g.:
> >
> > Peer 1:                            Peer 2:
> >   Vert_1                              Vert_2
> > (Total_V = 2)                  (Total_V = 2)
> > addVertex()
> > (Total_V = 3)
> >                                          getNumberOfV() => 2
> >
> > ------------------------ Sync ------------------------
> >
> >                                          getNumberOfV() => 3
> >
> >
> > Is there something like global counters or shared memory that it can
> address this issue?
> >
> > P.S. I have a small feeling that we don't need to track the total amount
> of vertices because vertex centered algorithms rarely need total numbers,
> they only depend on neighbors (I might be wrong though).
> >
> > Thanks,
> > Anastasis
>

--047d7b6da8c203f59104e188251a--