Return-Path: X-Original-To: apmail-incubator-giraph-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-giraph-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2B8CB90D5 for ; Sat, 1 Oct 2011 18:10:34 +0000 (UTC) Received: (qmail 11871 invoked by uid 500); 1 Oct 2011 18:10:33 -0000 Delivered-To: apmail-incubator-giraph-user-archive@incubator.apache.org Received: (qmail 11834 invoked by uid 500); 1 Oct 2011 18:10:33 -0000 Mailing-List: contact giraph-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: giraph-user@incubator.apache.org Delivered-To: mailing list giraph-user@incubator.apache.org Received: (qmail 11797 invoked by uid 99); 1 Oct 2011 18:10:33 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Oct 2011 18:10:33 +0000 Received: from localhost (HELO achingmbp15.local) (127.0.0.1) (smtp-auth username aching, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Oct 2011 18:10:33 +0000 Message-ID: <4E875798.7060706@apache.org> Date: Sat, 01 Oct 2011 11:10:32 -0700 From: Avery Ching User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: giraph-user@incubator.apache.org CC: Jake Mannix Subject: Re: On pre/post Application/Superstep contract References: <4E8603F4.6020803@apache.org> In-Reply-To: Content-Type: multipart/alternative; boundary="------------060605000901090807050906" This is a multi-part message in MIME format. --------------060605000901090807050906 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Can you show me an example of the inner Context class idea? Sounds interesting... Another question is whether to have the (pre|post)(Application|Superstep)() methods executed one as an aggregate and passed to the workers, or executed per worker. I think the former might be a little expensive, depending on how big the "Context" is. Perhaps executed per worker makes the most sense. Any other thoughts? Maybe aggregator methods would be useful as well, say to do this like write the aggregators for the entire application every now and then. That would probably get executed on the master. I think the current uses of the (pre|post)(Application|Superstep)() methods are fine in the per-worker specific way of thinking. Avery On 10/1/11 7:06 AM, Jake Mannix wrote: > On Sat, Oct 1, 2011 at 2:29 AM, Hyunsik Choi > wrote: > > Now, that way looks good. Probably, later we could improve that > like Context > of MapReduce. > > > ooooooh! I really that suggestion, actually. If every BasicVertex has an > inner Context class, we can allow user applications to define/extend their > Context and we can avoid even doing any of this setClass() and reflection > based stuff, if we do it right. Typesafe context object FTW! > > -jake > > > -- > Hyunsik Choi > Database Lab, Korea University > > On Sat, Oct 1, 2011 at 3:01 AM, Avery Ching > wrote: > > It isn't visible (purposefully) since it is internal state. > > > > That being said, I believe this type of functionality would be > useful. > > Right now there is a lot of ugly static variables stored in Vertex > > implementations because of it. Perhaps we should add another > method in > > GiraphJob > > > > final public void setWorkerObjectClass(Class > > workerObjectClass); > > > > Then in BasicVertex > > > > public void preApplication(Configurable workerObject); > > public void postApplication(Configurable workerObject); > > public void preSuperstep(Configurable workerObject); > > public void postSuperstep(Configurable workerObject); > > public Configurable getWorkerObject(); > > > > Anyone else think of a cleaner way to do it? > > > > Avery > > > > On 9/30/11 8:42 AM, Claudio Martella wrote: > >> > >> afaik getGraphState() is not visible to my object. Or? > >> > >> On Fri, Sep 30, 2011 at 5:23 PM, Jake > Mannix> > >> wrote: > >>> > >>> Remember that there's already a "singleton"-like object > available to all > >>> vertices: the GraphState object, which has a handle on the > GraphMapper. > >>> Maybe this is the right place to get your handle on the > >>> FSDataOutputStream? > >>> -jake > >>> On Fri, Sep 30, 2011 at 7:25 AM, Claudio Martella > >>> > wrote: > >>>> > >>>> Hello, > >>>> > >>>> to my understanding pre/post Application/Superstep methods > are called > >>>> ONCE on a "fake" vertex on each worker (the so called > >>>> representativeVertex). This means that these methods should > not depend > >>>> on any specific-vertex data. > >>>> > >>>> As I'm trying to sort out my Emitter, I thought I could > create one > >>>> FSDataOutputStream per worker which each Vertex belonging to that > >>>> worker could share (which would be even thread-safe as each > worker is > >>>> not parallel). > >>>> > >>>> The questions are: > >>>> > >>>> 1) how to share the FSDataOutputFormat object created at > >>>> preApplication() (and closed at postApplication()) created by > this > >>>> representativeVertex? > >>>> > >>>> 2) about the filename, I'd be happy to have access to the > Worker Id so > >>>> to create an outputfile filename as with happens with > reducers and > >>>> part files by FileOutputFormat > (i.e.-workerid). > >>>> > >>>> > >>>> The "best" idea i have in my mind right now is to use the calling > >>>> vertex (the representativeVertex) hashCode as the id, and > create an > >>>> external Singleton where i can request register and request the > >>>> outputfiles similarly to what happens with Aggregators now, > and by > >>>> passing the *this* reference as an index to this map. Any > better idea? > >>>> :) > >>>> > >>>> > >>>> -- > >>>> Claudio Martella > >>>> claudio.martella@gmail.com > >>> > >> > >> > > > > > > --------------060605000901090807050906 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Can you show me an example of the inner Context class idea?  Sounds interesting...

Another question is whether to have the (pre|post)(Application|Superstep)() methods executed one as an aggregate and passed to the workers, or executed per worker.  I think the former might be a little expensive, depending on how big the "Context" is.  Perhaps executed per worker makes the most sense.  Any other thoughts?

Maybe aggregator methods would be useful as well, say to do this like write the aggregators for the entire application every now and then.  That would probably get executed on the master.  I think the current uses of the (pre|post)(Application|Superstep)() methods are fine in the per-worker specific way of thinking.

Avery

On 10/1/11 7:06 AM, Jake Mannix wrote:
On Sat, Oct 1, 2011 at 2:29 AM, Hyunsik Choi <hyunsik@apache.org> wrote:
Now, that way looks good. Probably, later we could improve that like Context
of MapReduce.

ooooooh!  I really that suggestion, actually.  If every BasicVertex has an
inner Context class, we can allow user applications to define/extend their
Context and we can avoid even doing any of this setClass() and reflection
based stuff, if we do it right.  Typesafe context object FTW!

  -jake 
 

--
Hyunsik Choi
Database Lab, Korea University

On Sat, Oct 1, 2011 at 3:01 AM, Avery Ching <aching@apache.org> wrote:
> It isn't visible (purposefully) since it is internal state.
>
> That being said, I believe this type of functionality would be useful.
>  Right now there is a lot of ugly static variables stored in Vertex
> implementations because of it.  Perhaps we should add another method in
> GiraphJob
>
> final public void setWorkerObjectClass(Class<? extends Configurable>
> workerObjectClass);
>
> Then in BasicVertex
>
> public void preApplication(Configurable workerObject);
> public void postApplication(Configurable workerObject);
> public void preSuperstep(Configurable workerObject);
> public void postSuperstep(Configurable workerObject);
> public Configurable getWorkerObject();
>
> Anyone else think of a cleaner way to do it?
>
> Avery
>
> On 9/30/11 8:42 AM, Claudio Martella wrote:
>>
>> afaik getGraphState() is not visible to my object. Or?
>>
>> On Fri, Sep 30, 2011 at 5:23 PM, Jake Mannix<jake.mannix@gmail.com>
>>  wrote:
>>>
>>> Remember that there's already a "singleton"-like object available to all
>>> vertices: the GraphState object, which has a handle on the GraphMapper.
>>> Maybe this is the right place to get your handle on the
>>> FSDataOutputStream?
>>>   -jake
>>> On Fri, Sep 30, 2011 at 7:25 AM, Claudio Martella
>>> <claudio.martella@gmail.com>  wrote:
>>>>
>>>> Hello,
>>>>
>>>> to my understanding pre/post Application/Superstep methods are called
>>>> ONCE on a "fake" vertex on each worker (the so called
>>>> representativeVertex). This means that these methods should not depend
>>>> on any specific-vertex data.
>>>>
>>>> As I'm trying to sort out my Emitter, I thought I could create one
>>>> FSDataOutputStream per worker which each Vertex belonging to that
>>>> worker could share (which would be even thread-safe as each worker is
>>>> not parallel).
>>>>
>>>> The questions are:
>>>>
>>>> 1) how to share the FSDataOutputFormat object created at
>>>> preApplication() (and closed at postApplication()) created by this
>>>> representativeVertex?
>>>>
>>>> 2) about the filename, I'd be happy to have access to the Worker Id so
>>>> to create an outputfile filename as with happens with reducers and
>>>> part files by FileOutputFormat (i.e.<userdefinedfilename>-workerid).
>>>>
>>>>
>>>> The "best" idea i have in my mind right now is to use the calling
>>>> vertex (the representativeVertex) hashCode as the id, and create an
>>>> external Singleton where i can request register and request the
>>>> outputfiles similarly to what happens with Aggregators now, and by
>>>> passing the *this* reference as an index to this map. Any better idea?
>>>> :)
>>>>
>>>>
>>>> --
>>>>     Claudio Martella
>>>>     claudio.martella@gmail.com
>>>
>>
>>
>
>


--------------060605000901090807050906--