Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 96ED2119F4 for ; Thu, 28 Aug 2014 18:26:54 +0000 (UTC) Received: (qmail 40472 invoked by uid 500); 28 Aug 2014 18:26:54 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 40421 invoked by uid 500); 28 Aug 2014 18:26:54 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 40410 invoked by uid 99); 28 Aug 2014 18:26:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Aug 2014 18:26:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of saltzm@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Aug 2014 18:26:28 +0000 Received: by mail-wi0-f182.google.com with SMTP id z2so1378302wiv.9 for ; Thu, 28 Aug 2014 11:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=kZs4f+JyYRn5IAl2hUnbRjbyBQMdti/tLwhn0EE08wM=; b=cQPE6p96Dqj9Qj8oqFwUhd5tMtcEKvDcmKqr+9iS36ZO/nIY92K5EA+AnmuIXJXtNZ aMPByOtpjbD5PtZYinVV8XYuYrs/oVr4W8YGZrBIG6UZ9xEpCcm8fCBBNTqHamgTf04Q V3ZT1ecP5Nyx0xXxNH7+rQPFdxCpmqvNUjJpKGj3n5uhmfR5TwWKn1oUpwTe7Ru00opl zhlTSxi1425glF8Vu84wkYcWKC1waCjkrMSQk/bKTAoUl/1okCRqMWDLvK59iLeSO/Ia il41WKbtdSrEfYw1l8RozBQKz2OckJr5FZvEn2Kfz0PNGt19lB4liAAJXSz0Ky40EAbx BjQw== MIME-Version: 1.0 X-Received: by 10.180.24.35 with SMTP id r3mr8044475wif.71.1409250386874; Thu, 28 Aug 2014 11:26:26 -0700 (PDT) Received: by 10.194.239.225 with HTTP; Thu, 28 Aug 2014 11:26:26 -0700 (PDT) In-Reply-To: References: Date: Thu, 28 Aug 2014 20:26:26 +0200 Message-ID: Subject: Re: How do I validate customArguments? From: Matthew Saltz To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=f46d043bdf20b5b36b0501b4ac12 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043bdf20b5b36b0501b4ac12 Content-Type: text/plain; charset=UTF-8 Matt, I'm not sure if you've resolved this problem already or not, but if you haven't: The initialize() method isn't limited to registering aggregators, and in fact, in my project I use it to do exactly what you're describing to check and load custom configuration parameters. Inside the initialize() method, I do this: * String numPreprocessingStepsConf = getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);* * numPreprocessingSteps = (numPreprocessingStepsConf != null) ?* * Integer.parseInt(numPreprocessingStepsConf.trim()) :* * DEFAULT_NUMBER_OF_PREPROCESSING_STEPS;* * System.out.println("Number of preprocessing steps: " + numPreprocessingSteps);* where at the class level I declare: public static final String NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT = "wcc.numPreprocessingSteps"; public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS = 1; public static int numPreprocessingSteps; To set the property, I use the option "-ca wcc.numPreprocessingSteps=". If you need to check that it's properly formatted and not store them, this is a fine place to do it as well, given that it's run before the input superstep (see the giraph code in BspServiceMaster, line 1617 in the stable 1.1.0 release). What happens is that on the master, the MasterThread calls coordinateSuperstep() on a BspServiceMaster object, which checks if it's the input superstep, and if so, calls initialize() on the MasterCompute object (created in the becomeMaster() method of BspServiceMaster). Hope this helps, Matthew On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell wrote: > Hi again. My application needs to pass in a String argument to the > computation which each Vertex needs access to. (The argument is a list of > the form "[item1, item2, ...]".) I found --customArguments (which I set in > my tests via conf.set(, )) but I need to check that it's > properly formatted. Where do I do that? The only thing I thought of is to > specify a DefaultMasterCompute subclass whose initialize() does the check, > but all the initialize() examples do is register aggregators; none of them > check args or do anything else. Thanks in advance! -- matt > > -- > Matthew Cornell | matt@matthewcornell.org | 413-626-3621 | 34 Dickinson > Street, Amherst MA 01002 | matthewcornell.org > --f46d043bdf20b5b36b0501b4ac12 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Matt,

I'm not sure if you've re= solved this problem already or not, but if you haven't: The initialize(= ) method isn't limited to registering aggregators, and in fact, in my p= roject I use it to do exactly what you're describing to check and load = custom configuration parameters. Inside the initialize() method, I do this:=

=C2=A0 =C2=A0 String numPreprocessingSte= psConf =3D getConf().get(NUMBER_OF_PREPROCESSING_STEPS_CONF_OPT);
=C2=A0 =C2=A0 numPreprocessingSteps =3D (numPreprocessingStepsCon= f !=3D null) ?
<= b style=3D"background-color:rgb(255,255,255)">=C2=A0 =C2=A0 =C2=A0 =C2=A0 I= nteger.parseInt(numPreprocessingStepsConf.trim()) :
=C2=A0 =C2=A0 =C2=A0 =C2=A0 DEFAULT_NUMBER_OF_PREPROCESSING_= STEPS;
=C2=A0 =C2=A0 System.out.println(&qu= ot;Number of preprocessing steps: " + numPreprocessingSteps);

where at the class level I declare:

=C2=A0 public static final String NUMBER_OF_PREPRO= CESSING_STEPS_CONF_OPT =3D "wcc.numPreprocessingSteps";
=C2=A0 public static final int DEFAULT_NUMBER_OF_PREPROCESSING_STEPS =3D 1;=
=C2=A0 public static int numPreprocessingSteps;

To set the prope= rty, I use the option "-ca wcc.numPreprocessingSteps=3D<number of s= teps I want>". If you need to check that it's properly formatte= d and not store them, this is a fine place to do it as well, given that it&= #39;s run before the input superstep (see the giraph code in BspServiceMast= er, line 1617 in the stable 1.1.0 release). What happens is that on the mas= ter, the MasterThread calls coordinateSuperstep() on a BspServiceMaster obj= ect, which checks if it's the input superstep, and if so, calls initial= ize() on the MasterCompute object (created in the becomeMaster() method of = BspServiceMaster).=C2=A0

Hope this helps,
M= atthew



On Tue, Aug 26, 2014 at 4:36 PM, Matthew Cornell <matt@mat= thewcornell.org> wrote:
Hi again. My application ne= eds to pass in a String argument to the computation which each Vertex needs= access to. (The argument is a list of the form "[item1, item2, ...]&q= uot;.) I found --customArguments (which I set in my tests via conf.set(<= arg_name>, <arg_val>)) but I need to check that it's properly = formatted. Where do I do that? The only thing I thought of is to specify a = DefaultMasterCompute subclass whose initialize() does the check, but all th= e initialize() examples do is register aggregators; none of them check args= or do anything else. Thanks in advance! -- matt

--
Matthew Cornell | matt@matthewcornell.org | 413-626-3621= | 34 Dickinson Street, Amherst MA 01002 | matthewcornell.org

--f46d043bdf20b5b36b0501b4ac12--