Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E691911FA3 for ; Wed, 10 Sep 2014 13:16:12 +0000 (UTC) Received: (qmail 57426 invoked by uid 500); 10 Sep 2014 13:16:12 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 57376 invoked by uid 500); 10 Sep 2014 13:16:12 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 57366 invoked by uid 99); 10 Sep 2014 13:16:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Sep 2014 13:16:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthewcornell@gmail.com designates 209.85.216.175 as permitted sender) Received: from [209.85.216.175] (HELO mail-qc0-f175.google.com) (209.85.216.175) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Sep 2014 13:16:08 +0000 Received: by mail-qc0-f175.google.com with SMTP id c9so19248150qcz.34 for ; Wed, 10 Sep 2014 06:15:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=TNBNdNOo1aH4eVTeE8CNVYN7+aMPmEe2FWuCwFtIKbs=; b=IIgrM8FqNm0Up1ZNzW9QMIk5kiJqr/lDGx95tXB+oNnrchQRsr00zFdg0GvwsgrIT8 Fh3A0/eEsSQHZ715ZF3Yu4Fi6cUSI7ZVVaZg21u1XOeUVP/ZDEIbR1cSf2zSyX4tALzL jWSYO9Swc+lUHf1AlINU4ykavxZ763jL6elqypd+xPbwJZ2VpQ7pVtkr4tj30hjWgZhN OrwQlY5Bo7CzJPLNwJ9fayv4FUq2FlxI4JgFOcqp5ACyRAobwuJ5KGN55e+HECLPXIIO hHgmnj4lEcSaSQ79kjMsBYx3KNF80H2lPijslEXK+0Bwr9OsW4Z6cgV5f8DKPqX18ypc Awow== X-Received: by 10.229.131.5 with SMTP id v5mr59512747qcs.21.1410354946791; Wed, 10 Sep 2014 06:15:46 -0700 (PDT) MIME-Version: 1.0 Sender: matthewcornell@gmail.com Received: by 10.224.28.134 with HTTP; Wed, 10 Sep 2014 06:15:06 -0700 (PDT) In-Reply-To: References: From: Matthew Cornell Date: Wed, 10 Sep 2014 09:15:06 -0400 X-Google-Sender-Auth: ZPvJ_2ki_IJt0_kqXl8e-6MFpWc Message-ID: Subject: Re: How do I validate customArguments? To: user Content-Type: multipart/alternative; boundary=001a1132e2b69c62370502b5d93e X-Virus-Checked: Checked by ClamAV on apache.org --001a1132e2b69c62370502b5d93e Content-Type: text/plain; charset=UTF-8 Sorry for the long delay, Matthew. That's really helpful. Right now I'm stuck on apparently running out of memory on our little cluster, but the log messages are confusing. I'm putting together a question, but in the meantime I'll try one of the simpler examples such as degree count to see if /anything/ will run against my graph, which is very small (100K and edges nodes). -- matt On Thu, Aug 28, 2014 at 2:26 PM, Matthew Saltz wrote: > Matt, > > I'm not sure if you've resolved this problem already or not, but if you > haven't: The initialize() method isn't limited to registering aggregators, > and in fact, in my project I use it to do exactly what you're describing to > check and load custom configuration parameters. Inside the initialize() > method, I do this: > > * String numPreprocessingStepsConf = > getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);* > * numPreprocessingSteps = (numPreprocessingStepsConf != null) ?* > * Integer.parseInt(numPreprocessingStepsConf.trim()) :* > * DEFAULT_NUMBER_OF_PREPROCESSING_STEPS;* > * System.out.println("Number of preprocessing steps: " + > numPreprocessingSteps);* > > where at the class level I declare: > > public static final String NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT = > "wcc.numPreprocessingSteps"; > public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS = 1; > public static int numPreprocessingSteps; > > To set the property, I use the option "-ca > wcc.numPreprocessingSteps=". If you need to check > that it's properly formatted and not store them, this is a fine place to do > it as well, given that it's run before the input superstep (see the giraph > code in BspServiceMaster, line 1617 in the stable 1.1.0 release). What > happens is that on the master, the MasterThread calls coordinateSuperstep() > on a BspServiceMaster object, which checks if it's the input superstep, and > if so, calls initialize() on the MasterCompute object (created in the > becomeMaster() method of BspServiceMaster). > > Hope this helps, > Matthew > > > > On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell > wrote: > >> Hi again. My application needs to pass in a String argument to the >> computation which each Vertex needs access to. (The argument is a list of >> the form "[item1, item2, ...]".) I found --customArguments (which I set in >> my tests via conf.set(, )) but I need to check that it's >> properly formatted. Where do I do that? The only thing I thought of is to >> specify a DefaultMasterCompute subclass whose initialize() does the check, >> but all the initialize() examples do is register aggregators; none of them >> check args or do anything else. Thanks in advance! -- matt >> >> -- >> Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson >> Street, Amherst MA 01002 | matthewcornell.org >> > > -- Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org --001a1132e2b69c62370502b5d93e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Sorry for the long delay, Matthew. That's really helpf= ul. Right now I'm stuck on apparently running out of memory on our litt= le cluster, but the log messages are confusing. I'm putting together a = question, but in the meantime I'll try one of the simpler examples such= as degree count to see if /anything/ will run against my graph, which is v= ery small (100K and edges nodes). -- matt

On Thu, Aug 28, 2014 at 2:26 PM, Matthew = Saltz <saltzm@gmail.com> wrote:
Matt,

I'm not sure if you'= ve resolved this problem already or not, but if you haven't: The initia= lize() method isn't limited to registering aggregators, and in fact, in= my project I use it to do exactly what you're describing to check and = load custom configuration parameters. Inside the initialize() method, I do = this:

=C2=A0 =C2=A0 String numPreprocessingSte= psConf =3D getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);
=C2=A0 =C2=A0 numPreprocessingSteps =3D (numPreprocessingStepsCon= f !=3D null) ?
<= b style=3D"background-color:rgb(255,255,255)">=C2=A0 =C2=A0 =C2=A0 =C2=A0 I= nteger.parseInt(numPreprocessingStepsConf.trim()) :
=C2=A0 =C2=A0 =C2=A0 =C2=A0 DEFAULT_NUMBER_OF_PREPROCESSING_= STEPS;
=C2=A0 =C2=A0 System.out.println(&qu= ot;Number of preprocessing steps: " + numPreprocessingSteps);

where at the class level I declare:

=C2=A0 public static final String NUMBER_OF_PREPRO= CESSING_STEPS_CONF_OPT =3D "wcc.numPreprocessingSteps";
=C2=A0 public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS =3D 1;=
=C2=A0 public static int numPreprocessingSteps;

To set the prope= rty, I use the option "-ca wcc.numPreprocessingSteps=3D<number of s= teps I want>". If you need to check that it's properly formatte= d and not store them, this is a fine place to do it as well, given that it&= #39;s run before the input superstep (see the giraph code in BspServiceMast= er, line 1617 in the stable 1.1.0 release). What happens is that on the mas= ter, the MasterThread calls coordinateSuperstep() on a BspServiceMaster obj= ect, which checks if it's the input superstep, and if so, calls initial= ize() on the MasterCompute object (created in the becomeMaster() method of = BspServiceMaster).=C2=A0

Hope this helps,
M= atthew



On Tue, Aug 26, 2014 at= 4:36 PM, Matthew Cornell <matt@matthewcornell.org> wr= ote:
Hi again. My application ne= eds to pass in a String argument to the computation which each Vertex needs= access to. (The argument is a list of the form "[item1, item2, ...]&q= uot;.) I found --customArguments (which I set in my tests via conf.set(<= arg_name>, <arg_val>)) but I need to check that it's properly = formatted. Where do I do that? The only thing I thought of is to specify a = DefaultMasterCompute subclass whose initialize() does the check, but all th= e initialize() examples do is register aggregators; none of them check args= or do anything else. Thanks in advance! -- matt

--
Matthew Cornell | matt@matthewcornell.org | 413-626-3621= | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org




--
Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson Stree= t, Amherst MA 01002 | matthewcornell.org
--001a1132e2b69c62370502b5d93e--